Merge Request search-as-you-type for GitLab Duo

Problem to solve

GitLab duo chat will soon work on adding issues/MRs for pinned context. This requires typing to find an issue/MR so that it can be selected.

Proposal

Allow MRs (then issues) to be searched as a user types by using search-as-you-type on Elasticsearch.

Elastic offers the https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html mapping and search option. The suggester stores text and its filterable context in memory so it's very fast for giving suggestions as you type.

Quick demo for issues

demo_search_as_you_type

Mapping:

Screenshot_2024-08-14_at_13.01.35

Screenshot_2024-08-14_at_13.03.03

Query:

GET gitlab-development-issues/_search
{
  "size": 0, 
  "_source": ["title_suggester"], 
  "suggest": {
    "title_suggester": {
      "prefix": "issue hy",
      "completion": {
        "field": "title_suggester",
        "size": 1
      }
    }
  }
}

Work required

  • Add a new field for the suggester with recommended analyzer and tokenizer

  • Populate the field by adding a Elastic backfill migration.

  • Write a suggester query that returns only the first suggested result. Follow the pattern for aggregations as it is similar. This is what I used for the demo:

    Gitlab::Elastic::Helper.default.client.search(index: "gitlab-development-issues", body: {size: 0, suggest: {title_sugg: {prefix: query, completion: {field: "title_suggester", size: 1}}}}).dig("suggest").first[1].first['options']&.map{|sug| "#{sug['text']} (##{sug['_source']['id']})"}&.first

  • We will also need to add contexts. This is a suggester's filtering mechanism. Easiest would be to add a context for project_id and make sure the request always sends a project_id. If we need visibility_level and other fields, those need to be added as well. Max 10 contexts are allowed and everything is stored in heap so we need to be careful but we also shouldn't suggest titles that the user doesn't have access to.

    • Limit the suggester to signed-in users, on a project level, behind a FF with project scope. We'll need to duplicate the filters logic to the contexts. This is the query on prod for my user with a project selected:
    • Screenshot_2024-08-19_at_11.18.08
    • If we add a context for target_project_id and merge_requests_access_level, and limit the users that could get suggestions to those who fit this condition, maybe that's good enough for MVP.

  • Expose this via an API endpoint, either by adding an option to the current project search API or by creating a new endpoint. The result can return the suggested text as well as any fields in the source document such as ID, etc. so we should chat to the task force about which fields they would like to use.

Edited by Madelein van Niekerk