Skip to content

Add number of shards setting

Dmitry Gruzd requested to merge 415815-add-number-of-shards-setting into master

What does this MR do and why?

This MR adds a new setting to configure the number of shards we use for indexing. Hard limit will stay 16, but customers will be able to decrease it. Inactive shards will not receive updates, but will still be drained. I believe that this is a quick, but very useful iteration.

Related to #415815 (closed)

Migrations

$ bin/rake db:migrate
main: == [advisory_lock_connection] object_id: 229060, pg_backend_pid: 12639
main: == 20230621074446 AddElasticsearchNumberOfShardsToApplicationSettings: migrating
main: -- add_column(:application_settings, :elasticsearch_worker_number_of_shards, :integer, {:null=>false, :default=>2})
main:    -> 0.0027s
main: == 20230621074446 AddElasticsearchNumberOfShardsToApplicationSettings: migrated (0.0069s)

main: == 20230621074611 UpdateElasticsearchNumberOfShardsInApplicationSettingsForGitlabCom: migrating
main: == 20230621074611 UpdateElasticsearchNumberOfShardsInApplicationSettingsForGitlabCom: migrated (0.0034s)

main: == [advisory_lock_connection] object_id: 229060, pg_backend_pid: 12639
ci: == [advisory_lock_connection] object_id: 229340, pg_backend_pid: 12641
ci: == [advisory_lock_connection] object_id: 229340, pg_backend_pid: 12641
embedding: == [advisory_lock_connection] object_id: 229440, pg_backend_pid: 12643
embedding: == [advisory_lock_connection] object_id: 229440, pg_backend_pid: 12643
$ bin/rake db:rollback:main STEP=2
main: == [advisory_lock_connection] object_id: 228780, pg_backend_pid: 11948
main: == 20230621074611 UpdateElasticsearchNumberOfShardsInApplicationSettingsForGitlabCom: reverting
main: == 20230621074611 UpdateElasticsearchNumberOfShardsInApplicationSettingsForGitlabCom: reverted (0.0039s)

main: == 20230621074446 AddElasticsearchNumberOfShardsToApplicationSettings: reverting
main: -- remove_column(:application_settings, :elasticsearch_worker_number_of_shards, :integer, {:null=>false, :default=>2})
main:    -> 0.0015s
main: == 20230621074446 AddElasticsearchNumberOfShardsToApplicationSettings: reverted (0.0058s)

main: == [advisory_lock_connection] object_id: 228780, pg_backend_pid: 11948

Screenshots or screen recordings

SCR-20230621-jqlv

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Dmitry Gruzd

Merge request reports