ActiveContext: use search_embedding_version during search

What does this MR do and why?

Part 3 of 4 of #534318 (closed):

  1. !187724 (merged): add migration helper for updating collection metadata
  2. !188535 (merged): use indexing_embedding_versions to generate embeddings during indexing
  3. This MR: use search_embedding_version during searching
  4. !188560 (merged): update docs to show how to use this workflow to manage embedding models

This MR:

  • Allows using a knn query as either
    • Using the automatic embedding model and field by passing content only: ActiveContext::Query.knn(content: "Your content here", limit: 5)
    • Passing target, vector to override: ActiveContext::Query.knn(target: "embeddings_1", vector: [0.1, 0.2, 0.3], limit: 5)
  • Pass user to Processor - we need the user in order to track usage for embedding generation
  • Updates processors to prefer target and vector or fallback to content and generating embeddings.

References

#534318 (closed)

See https://gitlab.com/gitlab-org/gitlab/-/blob/144ec77cd48f682118dbb05417bf688a17d79017/gems/gitlab-active-context/doc/how_to.md for how the workflow will work.

How to set up and validate locally

  1. Run through steps in https://gitlab.com/gitlab-org/gitlab/-/blob/144ec77cd48f682118dbb05417bf688a17d79017/gems/gitlab-active-context/doc/how_to.md#set-embedding-model
  2. Enqueue and execute some docs and note that they use the embedding model specified (tail AIGW logs) and stores it on the correct field.
  3. Execute a search:
ActiveContext::Query.knn(content: "Your content here", limit: 5)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #534318 (closed)

Edited by Madelein van Niekerk

Merge request reports

Loading