# Task 3: Search Groups Using Elasticsearch ## Context Replace the current database query (`Api.groups(search)`) with Elasticsearch queries to improve performance. This requires creating a query builder, search service, and updating the API/frontend. ## Implementation Details ### 1. Create Group Query Builder **File**: `ee/lib/search/elastic/group_query_builder.rb` Follow pattern from `work_item_query_builder.rb`: ```ruby module Search module Elastic class GroupQueryBuilder def initialize(search_term, current_user, params = {}) @search_term = search_term @current_user = current_user @params = params end def build { query: query_hash, sort: sort_order, size: @params[:per_page] || 20 } end private def query_hash { bool: { must: must_clauses, filter: filter_clauses } } end def must_clauses return [] if @search_term.blank? [{ multi_match: { query: @search_term, fields: ['name^3', 'full_name^2', 'path^2', 'description'], type: 'best_fields', fuzziness: 'AUTO' } }] end def filter_clauses [ visibility_filter, traversal_filter ].compact end def visibility_filter # Filter based on user permissions # Public groups: visibility_level = 20 # Internal groups: visibility_level = 10 (if user logged in) # Private groups: only if user has access end def traversal_filter # Filter by parent if specified return unless @params[:parent_id] { term: { parent_id: @params[:parent_id] } } end def sort_order case @params[:order_by] when 'name' [{ 'name.keyword' => { order: 'asc' } }] when 'similarity' ['_score'] else [{ updated_at: { order: 'desc' } }] end end end end end ``` ### 2. Create Group Search Results Class **File**: `ee/lib/search/elastic/group_search_results.rb` ```ruby module Search module Elastic class GroupSearchResults attr_reader :current_user, :query, :params def initialize(current_user, query, params = {}) @current_user = current_user @query = query @params = params end def objects(scope = nil) return Group.none unless use_elasticsearch? # Build ES query es_query = GroupQueryBuilder.new(query, current_user, params).build # Execute search response = client.search( index: index_name, body: es_query ) # Hydrate results from database group_ids = response.dig('hits', 'hits')&.map { |hit| hit['_source']['id'] } || [] Group.id_in(group_ids).preload(:route) end def formatted_count objects.count end private def use_elasticsearch? ::Feature.enabled?(:elasticsearch_group_search, current_user) && ::Gitlab::CurrentSettings.elasticsearch_search? end def index_name Search::Elastic::References::Group.index end def client Group.__elasticsearch__.client end end end end ``` ### 3. Update Groups API **File**: `lib/api/groups.rb` (or create `ee/lib/api/ee/groups.rb` override) Modify the groups search endpoint: ```ruby get do if use_elasticsearch_for_groups? results = ::Search::Elastic::GroupSearchResults.new( current_user, params[:search], params ).objects present paginate(results), with: Entities::Group else # Existing database search groups = GroupsFinder.new(current_user, params).execute present paginate(groups), with: Entities::Group end end helpers do def use_elasticsearch_for_groups? ::Feature.enabled?(:elasticsearch_group_search, current_user) && ::Gitlab::CurrentSettings.elasticsearch_search? end end ``` ### 4. Update Frontend API Client **File**: `app/assets/javascripts/api/groups_api.js` The existing `Api.groups()` call should work without changes if the backend API is updated. The frontend just calls the API endpoint. ### 5. Add Feature Flag **File**: `config/feature_flags/development/elasticsearch_group_search.yml` ```yaml --- name: elasticsearch_group_search introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/XXXXXX rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/XXXXXX milestone: 'X.Y' type: development group: group::global search default_enabled: false ``` ### 6. Visibility and Permissions The query builder must respect: - Public groups (visibility_level: 20) - visible to everyone - Internal groups (visibility_level: 10) - visible to logged-in users - Private groups (visibility_level: 0) - visible only to members - Use `traversal_ids` for hierarchy-based filtering ## Files to Create/Modify - `ee/lib/search/elastic/group_query_builder.rb` (new) - `ee/lib/search/elastic/group_search_results.rb` (new) - `lib/api/groups.rb` or `ee/lib/api/ee/groups.rb` (modify) - `config/feature_flags/development/elasticsearch_group_search.yml` (new) ## Testing - `ee/spec/lib/search/elastic/group_query_builder_spec.rb` - Test text search on name, path, description - Test visibility filtering - Test parent/hierarchy filtering - Test sort orders - `ee/spec/lib/search/elastic/group_search_results_spec.rb` - Test result hydration - Test empty results - Test permission filtering - `ee/spec/requests/api/groups_spec.rb` (modify) - Test ES-powered search - Test fallback to database search - Test feature flag toggle ## Validation Steps 1. Enable feature flag: ```ruby Feature.enable(:elasticsearch_group_search) ``` 2. Test search via console: ```ruby results = Search::Elastic::GroupSearchResults.new( User.first, 'gitlab', {} ).objects ``` 3. Test via API: ```bash curl -H "Authorization: Bearer TOKEN" \ "http://localhost:3000/api/v4/groups?search=gitlab" ``` 4. Verify ES query: ```bash # Enable ES query logging curl -s "localhost:9200/gitlab-development-groups/_search?pretty" \ -d '{ "query": { "multi_match": { "query": "gitlab", "fields": ["name^3", "full_name^2"] }}}' ``` ## Performance Testing - Benchmark database query vs ES query - Test with 10k, 100k, 1M groups - Measure response time improvement - Test concurrent searches ## Acceptance Criteria - [ ] Group query builder constructs proper ES queries - [ ] Search results class executes ES searches and hydrates AR objects - [ ] API endpoint uses ES when feature flag enabled - [ ] Search respects visibility levels and user permissions - [ ] Feature flag controls ES vs DB search - [ ] Fuzzy matching works correctly - [ ] Sort options (similarity, name, date) work - [ ] Performance improvement measured and documented - [ ] Fallback to database search when ES unavailable