Skip to content

Switch workitem index to use new embedding field and model

What does this MR do and why?

Part 4 of 5 of Vertex text embedding model discontinuation on ... (#521836 - closed):

  1. Allow model to be specified: MR
  2. Add text-embedding-005 model to Vertex supported models on AI Gateway: MR
  3. Add embedding_1 field to work items index: MR
  4. Use embedding_1 field and new model during indexing, backfill embedding_1 and switch queries to use embedding_1 and new model 👈 this MR
  5. Nullify embedding_0 field: MR

Changes:

  • Adds the new model to MODEL_VERSIONS
  • Add embedding_1 to as_indexed_json once the migration in !187493 (merged) is complete. This field uses the new model
  • Add a backfill migration to backfill the embedding_1 field
  • Once the backfill ^ migration is complete:
    • Use the embedding_1 field and model during searching
    • Stop adding embedding_0 field in as_indexed_json

embedding1_demo

References

Vertex text embedding model discontinuation on ... (#521836 - closed)

https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings-api#model_versions

How to set up and validate locally

  1. Update ee/lib/search/elastic/work_item_query_builder.rb to return true in hybrid_work_item_search?
  2. Update ee/elastic/migrate/20250409120010_backfill_work_items_embeddings1.rb to make PROJECT_IDS something you have in your db
  3. Run migration worker and bookkeeping service on repeat
Elastic::MigrationWorker.new.perform
Elastic::ProcessBookkeepingService.new.execute
Elastic::ProcessInitialBookkeepingService.new.execute
::Search::Elastic::ProcessEmbeddingBookkeepingService.new.execute
  1. Note that the embedding_1 is now present in the mapping
  2. Note that embedding_1 field is present, embedding_0 is not
  3. Hybrid search still works
  4. Optional: look at the AI Gateway logs and see that the vertex endpoint is now v1/proxy/vertex-ai/v1/projects/PROJECT/locations/LOCATION/publishers/google/models/text-embedding-005%3Apredict

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #521836 (closed)

Edited by Madelein van Niekerk

Merge request reports

Loading