Skip to content

Add smaller search curation thresholds for development environment

John Mason requested to merge jm-ilm-development-settings into master

What does this MR do and why?

This lowers the threshold for search index curation to occur in local development environments. The IndexCurationWorker will pick this up or curation can be triggered manually via a console.

Screenshots or screen recordings

Here is the output of GET _cat/indices?v after running ./bin/rake gitlab:elastic:index then triggering search index curation manually:

health status index                                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   gitlab-development-users-20221209-0030          sz9aFU8FQL28YCTUzy0RZw   5   1         47            0     65.4kb         65.4kb
yellow open   gitlab-development-notes-20221209-0030          BD32bj07RSKd_0gbKi7etA   5   1        937            0    172.3kb        172.3kb
yellow open   gitlab-development-notes-20221209-0031          wzNTVUgvQHSRLuiWzUcsEA   5   1          0            0       450b           450b
yellow open   gitlab-development-merge_requests-20221209-0031 zn055CQURPOqTgoKXsS54A   5   1          0            0       450b           450b
yellow open   gitlab-development-migrations                   CsSM_OwrTXC7cYf_0Z7Fhg   1   1         26            0      5.9kb          5.9kb
yellow open   gitlab-development-merge_requests-20221209-0030 zlImsYhdR6edn5iYmwHYug   5   1        141            0    105.2kb        105.2kb
yellow open   gitlab-development-issues-20221209-0031         3QAliytFRviEFPCDlfpxYg   5   1          0            0       450b           450b
yellow open   gitlab-development-issues-20221209-0030         TYs-U2VSReC7nilYaunZQA   5   1        461            0    155.6kb        155.6kb
yellow open   gitlab-development-commits-20221209-0030        KuiPSXuVSw-ejgcw6kAe8Q   5   1          0            0        1kb            1kb
yellow open   gitlab-development-20221209-0030                bzQ0txoYSquqtimT6WgJ5Q   5   1         98            0      116kb          116kb

How to set up and validate locally

Inspect thresholds are lower for development environment

[1] pry(main)> Gitlab::Search::IndexCurator::DEFAULT_SETTINGS
=> {:dry_run=>true,
 :debug=>false,
 :force=>false,
 :max_shard_size_gb=>1,
 :max_docs_denominator=>100,
 :min_docs_before_rollover=>50,
 :max_docs_shard_count=>5,
 :ignore_patterns=>[/migrations/],
 :include_patterns=>[],
 :index_pattern=>"gitlab*"}

Run rollover manually

Gitlab::Search::IndexCurator.curate(dry_run: false)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by John Mason

Merge request reports