Add further validations to embedding model update

What does this MR do and why?

In preparation for the Embedding Model Selection UI, this MR adds further validations when updating an embedding model via the ::Ai::ActiveContext::EmbeddingModelActivationService:

  • if there is a current model, validates that the model update params introduces changes to model_type, model_ref, or dimensions
    • we don't want to kick off a backfill process if there were actually no changes to the embedding model
  • validates that the model_ref is in the list defined in the MODELS_LOOKUP constant
    • unlike model_type and dimensions, the allowed model_ref can grow to a large list and will be very different for self-hosted models, so we don't incorporate this in the json validation schema
  • add specs to the ::Ai::ActiveContext::Embedding class to test that the defined MODEL_TYPES and EMBEDDING_DIMENSIONS match what's in the json validation schema

References

Screenshots or screen recordings

N/A

How to set up and validate locally

The unit tests should cover the validations needed, but we can verify that the EmbeddingModelActivationService still works without issues by testing the happy path:

# 1 - set a current model, and make sure next model is nil
Ai::ActiveContext::Collections::Code.collection_record.update_metadata!(
  current_indexing_embedding_model: {
    model_type: :gitlab_managed,
    model_ref: 'test_embedding_1',
    dimensions: 768,
    field: 'embeddngs_v1'
  },
  next_indexing_embedding_model: nil
)

# used for querying the tasks
time_before_creation = Time.current

# 2 - execute the service
::Ai::ActiveContext::EmbeddingModelActivationService.new(
  collection_class: Ai::ActiveContext::Collections::Code,
  model_ref: 'text_embedding_005_vertex',
  dimensions: 768,
  model_type: :gitlab_managed
).execute!

# 3 - check that the expected tasks are created
tasks = ::Ai::ActiveContext::Task.where('created_at > ?', time_before_creation).order(id: :asc)
puts tasks.map(&:name).join("\n")

# expected output
Ai::ActiveContext::Tasks::AddEmbeddingsField
Ai::ActiveContext::Tasks::BackfillEmbeddings
Ai::ActiveContext::Tasks::UpdateCollectionMetadata
Ai::ActiveContext::Tasks::NullifyField

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #582638 (closed)

Edited by Pam Artiaga

Merge request reports

Loading