Switch workitem index to use new embedding field and model
What does this MR do and why?
Part 4 of 5 of Vertex text embedding model discontinuation on ... (#521836 - closed):
- Allow model to be specified: MR
- Add
text-embedding-005
model to Vertex supported models on AI Gateway: MR - Add
embedding_1
field to work items index: MR -
Use
embedding_1
field and new model during indexing, backfillembedding_1
and switch queries to useembedding_1
and new model👈 this MR - Nullify
embedding_0
field: MR
Changes:
- Adds the new model to
MODEL_VERSIONS
- Add
embedding_1
toas_indexed_json
once the migration in !187493 (merged) is complete. This field uses the new model - Add a backfill migration to backfill the
embedding_1
field - Once the backfill ^ migration is complete:
- Use the
embedding_1
field and model during searching - Stop adding
embedding_0
field inas_indexed_json
- Use the
References
Vertex text embedding model discontinuation on ... (#521836 - closed)
How to set up and validate locally
- Update
ee/lib/search/elastic/work_item_query_builder.rb
to returntrue
inhybrid_work_item_search?
- Update
ee/elastic/migrate/20250409120010_backfill_work_items_embeddings1.rb
to make PROJECT_IDS something you have in your db - Run migration worker and bookkeeping service on repeat
Elastic::MigrationWorker.new.perform
Elastic::ProcessBookkeepingService.new.execute
Elastic::ProcessInitialBookkeepingService.new.execute
::Search::Elastic::ProcessEmbeddingBookkeepingService.new.execute
- Note that the
embedding_1
is now present in the mapping - Note that
embedding_1
field is present,embedding_0
is not - Hybrid search still works
- Optional: look at the AI Gateway logs and see that the vertex endpoint is now
v1/proxy/vertex-ai/v1/projects/PROJECT/locations/LOCATION/publishers/google/models/text-embedding-005%3Apredict
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #521836 (closed)
Edited by Madelein van Niekerk