Skip to content

Add embeddings to issues if on Elasticsearch 8+

Madelein van Niekerk requested to merge 454251-add-embeddings-to-issues into master

What does this MR do and why?

Creates a new Elastic migration which adds a vector field to store embeddings in the issues index.

This migration is skippable. It will only be executed if the platform is elasticsearch and the version is 8+.

As per this thread, the initial implementation will not used quantized vectors but once this is available on the AI Gateway, we'll need another migration to update the mappings.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Supported platform and version

curl http://localhost:9200                                
{
  ...
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "8.11.4",
Elastic::MigrationWorker.new.perform
MigrationWorker: migration[AddEmbeddingToIssues] executing migrate method
[Elastic::Migration: 20240410193847] Adding {:embedding=\u003e{:type=\u003e\"dense_vector\", :dims=\u003e768, :similarity=\u003e\"dot_product\", :index=\u003etrue}} to gitlab-development-issues mapping
MigrationWorker: migration[AddEmbeddingToIssues] updating with completed: true
MigrationWorker: no migration available

Unsupported platform and version

curl http://localhost:9200                                
{
  ...
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "7.17.20"
  ...
Elastic::MigrationWorker.new.perform
MigrationWorker: no migration available

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Related to #454251 (closed)

Edited by Madelein van Niekerk

Merge request reports