Skip to content

Add Search Index Curation Worker

John Mason requested to merge jm-ilm-sidekiq-worker-373985 into master

What does this MR do and why?

Introduces a cron worker that will curate our indices based on the guidance described in https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html#tuning.

This is behind the feature flag search_index_curation: #375274

Part of epic &8746

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Create demo index
curl -XPOST "http://host.docker.internal:9200/_reindex" -H 'Content-Type: application/json' -d'
{
  "source": {
    "index": "gitlab-development-issues"
  },
  "dest": {
    "index": "search-curation-202210-0001"
  }
}'
  1. Create alias (notice it does not have is_write_index specififed. This is how aliases are currently.)
curl -XPUT "http://host.docker.internal:9200/search-curation/_aliases" -H 'Content-Type: application/json' -d'
{
  "actions": [
    {
      "add": {
        "alias": "search-curation",
        "index": "search-curation-202210-0001"
      }
    }
  ]
}'
  1. In rails console, run curator on search-curation index pattern
Gitlab::Search::IndexCurator.curate(index_pattern: 'search-curation*')
  1. Verify aliases are updated
curl -XGET "http://host.docker.internal:9200/_cat/aliases?v&s=index"
  1. Verify both indices have the same settings
curl -XGET "http://host.docker.internal:9200/search-curation-202210-0001/"
curl -XGET "http://host.docker.internal:9200/search-curation-202210-0002/"

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by John Mason

Merge request reports