ActiveContext: add indexing_embedding_fields to Collection class
What does this MR do and why?
- Adds the
indexing_embedding_fieldsmethod toCollection- this will replace the version-based
Collection.current_embedding_fieldswhen we switch to the new model design
- this will replace the version-based
- Updates the
ActiveContext::Databases::Elasticsearch::Clientto use bothCollection.current_embedding_fieldsand Collection.indexing_embedding_fields` when adding the embedding fields to the query- see explanation comment in the code
- Adds tests for the
MarkRepositoryAsReadyEventWorker, which makes use of thecurrent_embedding_fields- this worker is not yet using the new
indexing_embedding_fields, but we need to add tests around the embedding fields in preparation for ActiveContext: integrate new model design into ... (!222417 - merged)
- this worker is not yet using the new
Step-by-step changes summary
| MR | Status |
|---|---|
| Introduce the new hash/object-based models | |
Add indexing_embedding_fields to Collection class |
This MR |
Add embeddings_with_model_redesign preprocessor |
Ready for review |
| Integrate model redesign into Code Embeddings pipeline | Pending |
References
- Issue: [ActiveContext] Redesign how models are referen... (#588847)
- MR introducing the hash/object-based model design: ActiveContext embedding model redesign (!222416 - merged)
Screenshots or screen recordings
N/A
How to set up and validate locally
Since we can be sure that indexing_embedding_fields is still empty in production, and that it's not yet used anywhere outside of the ActiveContext::Databases::Elasticsearch::Client.add_source_fields method, the unit tests should cover the validation.
You may validate in the console that indexing_embedding_fields is indeed an empty array if current_indexing_embedding_model are next_indexing_embedding_model model metadatas are both nil.
-
Set the metadata values:
Ai::ActiveContext::Collections::Code.collection_record.update_metadata!(current_indexing_embedding_model: nil, next_indexing_embedding_model: nil) -
indexing_embedding_fieldsshould return an empty arrayAi::ActiveContext::Collections::Code.indexing_embedding_fields => []
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #588847
Edited by Pam Artiaga