Skip to content

Migration to add knn mappings to workitems index

Context

Create abstraction layer to support Elasticsear... (#454764) and Move embeddings from issues index to workitems ... (#476537) are being done together in this order:

  1. !163009
  2. !163946 👈 this MR
  3. !164059 (merged)
  4. #479776
  5. #479778
  6. #479777

What does this MR do and why?

This MR adds a migration to add the mapping for embeddings in the workitems index when the index already exists. The mapping is the same as in Draft: Allow Elasticsearch and OpenSearch speci... (!163009).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

Elasticsearch:

  1. Run the Migration worker on repeat: Elastic::MigrationWorker.new.perform
  2. See that the migration was executed by looking at the logs
  3. Note that the embedding_0 field exists in the workitems index
::Gitlab::Elastic::Helper.default.get_mapping(index_name: "gitlab-development-work_items")
=> {"archived"=>{"type"=>"boolean"},
 "assignee_id"=>{"type"=>"integer"},
 "author_id"=>{"type"=>"integer"},
 "confidential"=>{"type"=>"boolean"},
 "created_at"=>{"type"=>"date"},
 "description"=>{"type"=>"text", "analyzer"=>"code_analyzer"},
 "due_date"=>{"type"=>"date"},
 "embedding_0"=>{"type"=>"dense_vector", "dims"=>768, "index"=>true, "similarity"=>"cosine"},

(Optional) OpenSearch:

  1. Connect to opensearch
  2. Run the Migration worker on repeat: Elastic::MigrationWorker.new.perform
  3. See that the migration was executed by looking at the logs
  4. Note that the embedding_0 field exists in the workitems index
::Gitlab::Elastic::Helper.default.get_mapping(index_name: "gitlab-development-work_items")
=> {"archived"=>{"type"=>"boolean"},
 "assignee_id"=>{"type"=>"integer"},
 "author_id"=>{"type"=>"integer"},
 "confidential"=>{"type"=>"boolean"},
 "created_at"=>{"type"=>"date"},
 "description"=>{"type"=>"text", "analyzer"=>"code_analyzer"},
 "due_date"=>{"type"=>"date"},
 "embedding_0"=>{"type"=>"knn_vector", "dimension"=>768, "method"=>{"engine"=>"nmslib", "space_type"=>"cosinesimil", "name"=>"hnsw", "parameters"=>{"ef_construction"=>100, "m"=>16}}},

Related to #454764

Edited by Madelein van Niekerk

Merge request reports