Skip to content
Snippets Groups Projects

Migration to add knn mappings to workitems index

Merged Madelein van Niekerk requested to merge 454764-knn-mappings-migration into master

Context

Create abstraction layer to support Elasticsear... (#454764 - closed) and Move embeddings from issues index to workitems ... (#476537 - closed) are being done together in this order:

  1. !163009 (merged)
  2. !163946 (merged) :point_left: this MR
  3. !164059 (merged)
  4. #479776 (closed)
  5. #479778 (closed)
  6. #479777 (closed)

What does this MR do and why?

This MR adds a migration to add the mapping for embeddings in the workitems index when the index already exists. The mapping is the same as in Allow Elasticsearch and OpenSearch specific map... (!163009 - merged).

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

Elasticsearch:

  1. Run the Migration worker on repeat: Elastic::MigrationWorker.new.perform
  2. See that the migration was executed by looking at the logs
  3. Note that the embedding_0 field exists in the workitems index
::Gitlab::Elastic::Helper.default.get_mapping(index_name: "gitlab-development-work_items")
=> {"archived"=>{"type"=>"boolean"},
 "assignee_id"=>{"type"=>"integer"},
 "author_id"=>{"type"=>"integer"},
 "confidential"=>{"type"=>"boolean"},
 "created_at"=>{"type"=>"date"},
 "description"=>{"type"=>"text", "analyzer"=>"code_analyzer"},
 "due_date"=>{"type"=>"date"},
 "embedding_0"=>{"type"=>"dense_vector", "dims"=>768, "index"=>true, "similarity"=>"cosine"},

(Optional) OpenSearch:

  1. Connect to opensearch
  2. Run the Migration worker on repeat: Elastic::MigrationWorker.new.perform
  3. See that the migration was executed by looking at the logs
  4. Note that the embedding_0 field exists in the workitems index
::Gitlab::Elastic::Helper.default.get_mapping(index_name: "gitlab-development-work_items")
=> {"archived"=>{"type"=>"boolean"},
 "assignee_id"=>{"type"=>"integer"},
 "author_id"=>{"type"=>"integer"},
 "confidential"=>{"type"=>"boolean"},
 "created_at"=>{"type"=>"date"},
 "description"=>{"type"=>"text", "analyzer"=>"code_analyzer"},
 "due_date"=>{"type"=>"date"},
 "embedding_0"=>{"type"=>"knn_vector", "dimension"=>768, "method"=>{"engine"=>"nmslib", "space_type"=>"cosinesimil", "name"=>"hnsw", "parameters"=>{"ef_construction"=>100, "m"=>16}}},

Related to #454764 (closed)

Edited by Madelein van Niekerk

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading