Global Search data migrations: add storage requirements
What does this MR do?
Related to #281158 (closed)
Add a new migration option space_requirements!
. Setting that option in a migration will require space_required_bytes
method to be defined. This method must return the amount of space required to run the migration (in bytes). The migration worker will check against cluster_free_size_bytes
and log + halt the migration if enough space is not available. Update/add specs and documentation.
Implement the new migration option in the move Notes to a new index. This migration has already completed in production and we were able to see exactly how much space notes took related to the main index (notes space used is 0.31% of the main index size). The space requirements option was configured using that ratio with a 5 times multiplier buffer.
How to test
- create a new migration in /ee/elastic/migrate, I named mine
20210301150000_test_migration.rb
,migrate
andcompleted?
methods don't have to do anything and can return false. - add space_requirements! option at the top
- do not define
space_required_bytes
method on the migration - run
Elastic::MigrationWorker.perform_async
verify the migration worker throws an error in the sidekiq logs - open a rails console and run
Gitlab::Elastic::Helper.default.cluster_free_size_bytes
to find out how much space is free in the cluster - define a method
space_required_bytes
method in the migration that returns a number higher than the bytes from the step prior - run
Elastic::MigrationWorker.perform_async
verify the migration is halted in the elasticsearch logs
logs
{"severity":"INFO","time":"2021-04-19T19:33:40.860Z","correlation_id":"5bd7696d8aac577711d6f7264b01e4b7","message":"MigrationWorker: migration[TestMigration] checking free space in cluster. Required space 402 GB. Free space 309 GB."}
{"severity":"WARN","time":"2021-04-19T19:33:40.861Z","correlation_id":"5bd7696d8aac577711d6f7264b01e4b7","message":"MigrationWorker: migration[TestMigration] You should have at least 402 GB of free space in the cluster to run this migration. Please increase the storage in your Elasticsearch cluster."}
{"severity":"INFO","time":"2021-04-19T19:33:40.861Z","correlation_id":"5bd7696d8aac577711d6f7264b01e4b7","message":"MigrationWorker: migration[TestMigration] updating with halted: true"}
Note: to reset the migration you will need to unset the halted
state
curl --request PUT \
--url $ES_CLUSTER/gitlab-development-migrations/_doc/20210301150000 \
--data '{
"completed": false
"state": {},
"started_at": "2021-01-07T14:14:30.291Z"
}'
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
-
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides - [-] Database guides
-
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team