Skip to content

Allow setting the shard/replica separately for ES indexes (step 1)

Dmitry Gruzd requested to merge 296882-standalone-shard-replica-settings into master

What does this MR do?

This MR introduces a new table for storing shards and replicas count for each index and migrates existing shard/replica settings from application_settings to this table. We also adapt current advanced settings page to using this new table.

table schema
gitlabhq_development=# \d elastic_index_settings
                                           Table "public.elastic_index_settings"
       Column       |           Type           | Collation | Nullable |                      Default
--------------------+--------------------------+-----------+----------+----------------------------------------------------
 id                 | bigint                   |           | not null | nextval('elastic_index_settings_id_seq'::regclass)
 alias_name         | text                     |           | not null |
 number_of_replicas | smallint                 |           | not null | 1
 number_of_shards   | smallint                 |           | not null | 5
 created_at         | timestamp with time zone |           | not null |
 updated_at         | timestamp with time zone |           | not null |
Indexes:
    "elastic_index_settings_pkey" PRIMARY KEY, btree (id)
    "index_elastic_index_settings_on_alias_name" UNIQUE, btree (alias_name)
Check constraints:
    "check_c30005c325" CHECK (char_length(alias_name) <= 255)

#296882 (closed)

Rollout plan

  • Step 1: (we are here) Create a new table and migrate existing index settings and change the backend to allow using this new table instead of application_settings
  • Step 2: Change the backend and the UI to allow changing settings for all existing indices. Ignoring no longer used fields (elasticsearch_replicas, elasticsearch_shards), in the application_settings table.

Migrations

up
$ VERSION=20210317100520 rake db:migrate:up
== 20210317100520 CreateElasticIndexSettings: migrating =======================
-- create_table(:elastic_index_settings, {})
-- quote_column_name(:alias_name)
   -> 0.0000s
   -> 0.0168s
-- quote_table_name("check_c30005c325")
   -> 0.0000s
-- quote_table_name(:elastic_index_settings)
   -> 0.0000s
-- execute("ALTER TABLE \"elastic_index_settings\"\nADD CONSTRAINT \"check_c30005c325\" CHECK (char_length(\"alias_name\") <= 255)\n")
   -> 0.0013s
== 20210317100520 CreateElasticIndexSettings: migrated (0.0238s) ==============

$ VERSION=20210324131727 rake db:migrate:up
== 20210324131727 MigrateElasticIndexSettings: migrating ======================
== 20210324131727 MigrateElasticIndexSettings: migrated (0.1913s) =============
down
$ VERSION=20210324131727 rake db:migrate:down
== 20210324131727 MigrateElasticIndexSettings: reverting ======================
== 20210324131727 MigrateElasticIndexSettings: reverted (0.0036s) =============

$ VERSION=20210317100520 rake db:migrate:down
== 20210317100520 CreateElasticIndexSettings: reverting =======================
-- drop_table(:elastic_index_settings)
   -> 0.0075s
== 20210317100520 CreateElasticIndexSettings: reverted (0.0076s) ==============

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

Related to #296882 (closed)

Edited by Dmitry Gruzd

Merge request reports