Skip to content

Add hybrid search query for issues

Madelein van Niekerk requested to merge 440424/hybrid-issue-search into master

What does this MR do and why?

Adds a hybrid search query for issues search.

Can be called as:

SearchService.new(user, { scope: 'issues', search: 'search_term: 'foo', hybrid_similarity: 0.5 }).search_results.objects('issues')

Or in web project issue search: http://127.0.0.1:3000/search?group_id=24&nav_source=navbar&project_id=2&scope=issues&search=test&hybrid_similarity=0.5.

If embeddings are present in the index, the search term will be converted into an embedding and passed into a kNN query which is combined with the normal BM25 search query to get results from both. If embeddings aren't present or if the embedding endpoint is rate limited, the query only uses BM25 matching.

Requires the hybrid_similarity param to be passed as an option which determines the level of strictness for kNN matches. 0 means not strict (get all matches), 1 would give only 100% matching embeddings. The param is added to experiment with the similarity settings until we find a sweet spot. I suggest starting at 0.5.

Later we will fix hybrid_similarity to a useful level and could extend this to group and global issue search.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Start the app in SaaS mode (or comment out this line)
  2. Enable the following feature flags: hybrid_issue_search, ai_global_switch, elasticsearch_issue_embedding
  3. Create issue embeddings
  4. Perform a normal project-level issue-scoped search: http://127.0.0.1:3000/search?group_id=24&nav_source=navbar&project_id=2&scope=issues&search=test
    1. Note that the search query does not have a knn clause
  5. Perform a hybrid project-level issue-scoped search: http://127.0.0.1:3000/search?group_id=24&nav_source=navbar&project_id=2&scope=issues&search=test&hybrid_similarity=0.5
    1. Note that the search query has a knn clause with embeddings
    2. Play around with the hybrid_similarity setting and see that you get more results with a lower similarity.

Related to #440424

Edited by Madelein van Niekerk

Merge request reports