Add hybrid search query for issues
What does this MR do and why?
Adds a hybrid search query for issues search.
Can be called as:
SearchService.new(user, { scope: 'issues', search: 'search_term: 'foo', hybrid_similarity: 0.5 }).search_results.objects('issues')
Or in web project issue search: http://127.0.0.1:3000/search?group_id=24&nav_source=navbar&project_id=2&scope=issues&search=test&hybrid_similarity=0.5.
If embeddings are present in the index, the search term will be converted into an embedding and passed into a kNN query which is combined with the normal BM25 search query to get results from both. If embeddings aren't present or if the embedding endpoint is rate limited, the query only uses BM25 matching.
Requires the hybrid_similarity
param to be passed as an option which determines the level of strictness for kNN matches. 0 means not strict (get all matches), 1 would give only 100% matching embeddings. The param is added to experiment with the similarity settings until we find a sweet spot. I suggest starting at 0.5.
Later we will fix hybrid_similarity
to a useful level and could extend this to group and global issue search.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Start the app in SaaS mode (or comment out this line)
- Enable the following feature flags:
hybrid_issue_search
,ai_global_switch
,elasticsearch_issue_embedding
- Create issue embeddings
- Perform a normal project-level issue-scoped search: http://127.0.0.1:3000/search?group_id=24&nav_source=navbar&project_id=2&scope=issues&search=test
- Note that the search query does not have a knn clause
- Perform a hybrid project-level issue-scoped search: http://127.0.0.1:3000/search?group_id=24&nav_source=navbar&project_id=2&scope=issues&search=test&hybrid_similarity=0.5
- Note that the search query has a knn clause with embeddings
- Play around with the
hybrid_similarity
setting and see that you get more results with a lower similarity.
Related to #440424