Global Search Group - 15.6 Planning

Priorities for %15.6

Features

typefeature

Product Manager @JohnMcGuire

In the %15.6, our main focus is to continue working on the features that we started in prior milestones.

• Add search filtering by programming language.
  We have finished the backend development in %15.3. The backend logic is behind a feature flag currently and we plan to enable it globally in %15.5 or %15.6. We will start the frontend in %15.6

  • [Feature flag] Enable search_blobs_language_aggregation
  • Add language facet to blob search result page - Front End

• Add All Users to the Elasticsearch index to make user list faster.
  Currently, user information is not indexed in the Advanced Search. User searches still need to get the data from the database. It can be slow and the requests time out quite often. In %15.6, we will start resolving this issue. We will start from the backend logic.

• As part of Document the REST API in OpenAPI V2, we will be working on annotating Global Search APIs.

  • https://gitlab.com/gitlab-org/gitlab/-/issues/377021+
  • https://gitlab.com/gitlab-org/gitlab/-/issues/376985+

• (Stretch) Add Labels as a facet in Advanced Search for Issues.
  We started indexing labels for Issues in %15.4. In %15.5, we finished the backend work including Labels as a facet: Aggregations API for labels. In %15.6, we hope to start the frontend development. The frontend work is tracked in the following issue which may get refined when we start working on it.

  • Labels as a facet: Client side for issues

Next1-3 releases candidates

Maintenance

typemaintenance

Engineering Manager: @changzhengliu

• Index lifecycle management for Advanced Search indices.
  As stated in this epic, manual index management does not scale when it comes to an ever-increasing Elasticsearch index. The design of the first iteration can be found at gitlab-org&8746. In %15.5, we started to explore index lifecycle management. The work will continue in %15.6. Please check the following issues for more details and progress of the work

  • ILM Sidekiq worker
  • Adapt Rails indexing for ILM
  • Adapt gitlab-elasticsearch-indexer for ILM
  • Adapt Advanced Search migration framework to ILM
  • Rollout the ILM migration worker feature flag
  • Zero downtime indexing update to support multiple indices

• Global Search Error Budget
  As we are finishing the work in Proposal: Error Budget Improvement for Global Search, we will need to watch closely on our Error Budget. Currently, we have two major issues in our mind to look at in the next milestone.

  • A loading issue on the Global Search Error Budget dashboard
  • Add global_search_indexing to sidekiq service
  • Add ErrorRate kind to global search SLI library
  • (Stretch) We are still overspending our budget at the moment. We need to look into the causes once the work in Proposal: Error Budget Improvement for Global Search is completed

• Use Rally for Elasticsearch performance testing.
  As we were considering what would be our benchmarking toolkit to measure our performance, Rally stood out to be an ideal candidate to perform such tasks. In %15.6, we'd like to look into how to set up a custom Rally track for our benchmarking test for Elasticsearch performance.

  • Rally: Determine strategy for test data
  • Rally: determine workload for first test

• Potential Performance Improvement
  The urgency of this effort is driven by the Global Search Error Budget. While we will do root cause analysis on the violation, the following items are in our mind now which could help improve the performance.

  • Validate and fix any issues with shard size and document count in Advanced Search indexes.
    The documentation states to consider the number of documents in the index when determining shard size. We have identified that at least one index, Notes, did not follow the guidance. So, we will validate this and fix any potential issues.

• Outline limitations of special character use in search within the Global Search documentation We will look into the first iteration of this SUSImpacting work.

Next1-3 releases candidates

Bugs

typebug

Quality Engineer @ebanks

• Global Search should handle files with long lines in code search results without trying to render the whole line
• Global Search Results should not render HTML
• gitlab:elastic:index task does not check pause setting
• ActiveRecord::InvalidForeignKey: PG::ForeignKeyViolation: ERROR: insert or update on table "index_statuses" violates foreign key ...

Next1-3 releases candidates

UX Design

UX

Designer @nickbrandt

• Ability to filter out forks when searching code
• Provide indicator for multiple matches in code results
• Support multi-project selection for Projects filter
  • This one had some previous work done on it, but now that we are moving these filters to the left sidebar it opens up the ability to handle this a bit better (similar to what is shown here)

Kickoff video

https://youtu.be/sn1EEEnwPKw

CC: @dgruzd @terrichu @john-mason @tbulva @maddievn @sdungarwal