Skip to content

Add knn index setting for workitem index for opensearch clusters

What does this MR do and why?

In 17.5, we released a migration that adds the embedding field to workitems index for customers running OpenSearch and Elasticsearch.

During development of the knn query, we found that sometimes creating a new workitems index fails with this error when running OpenSearch:

Elasticsearch::Transport::Transport::Errors::BadRequest: [400] {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Cannot set modelId or method parameters when index.knn setting is false"}]

This does not happen on OpenSearch versions < 2.17.0 (latest is 2.17.1). This MR seems to have introduced it: https://github.com/opensearch-project/k-NN/pull/1957.

Our specs didn't catch this because they run on OpenSearch version 2.15.0.

To ensure we don't break migrations or indexing for OpenSearch customers running the latest version of OpenSearch, we'll add the knn setting to all OpenSearch customers.

This MR:

  • When OpenSearch is running:
    • Adds knn: true index setting for when the workitems index is created from scratch
    • Reindexes workitems index using an elastic migration so that the setting and mappings are applied
  • When Elasticsearch is running:
    • Keep the current migration that has the mapping change only but skip if opensearch is running

This MR will be backported.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

  1. Run OpenSearch
  2. Run elastic migration worker and ensure that there are no errors
  3. Ensure the workitems index has the knn: true index setting and embedding_0 mapping
  4. [Optional] delete the workitems index and recreate the index: ::Gitlab::Elastic::Helper.default.create_standalone_indices(target_classes: [WorkItem]) and ensure that there are no errors
  5. Ensure the workitems index has the knn: true index setting and embedding_0 mapping
Edited by Madelein van Niekerk

Merge request reports

Loading