Skip to content

Global Search data migrations: add storage requirements

What does this MR do?

Related to #281158 (closed)

Add a new migration option space_requirements!. Setting that option in a migration will require space_required_bytes method to be defined. This method must return the amount of space required to run the migration (in bytes). The migration worker will check against cluster_free_size_bytes and log + halt the migration if enough space is not available. Update/add specs and documentation.

Implement the new migration option in the move Notes to a new index. This migration has already completed in production and we were able to see exactly how much space notes took related to the main index (notes space used is 0.31% of the main index size). The space requirements option was configured using that ratio with a 5 times multiplier buffer.

How to test

  • create a new migration in /ee/elastic/migrate, I named mine 20210301150000_test_migration.rb, migrate and completed? methods don't have to do anything and can return false.
  • add space_requirements! option at the top
  • do not define space_required_bytes method on the migration
  • run Elastic::MigrationWorker.perform_async verify the migration worker throws an error in the sidekiq logs
  • open a rails console and run Gitlab::Elastic::Helper.default.cluster_free_size_bytes to find out how much space is free in the cluster
  • define a method space_required_bytes method in the migration that returns a number higher than the bytes from the step prior
  • run Elastic::MigrationWorker.perform_async verify the migration is halted in the elasticsearch logs

logs

{"severity":"INFO","time":"2021-04-19T19:33:40.860Z","correlation_id":"5bd7696d8aac577711d6f7264b01e4b7","message":"MigrationWorker: migration[TestMigration] checking free space in cluster. Required space 402 GB. Free space 309 GB."}
{"severity":"WARN","time":"2021-04-19T19:33:40.861Z","correlation_id":"5bd7696d8aac577711d6f7264b01e4b7","message":"MigrationWorker: migration[TestMigration] You should have at least 402 GB of free space in the cluster to run this migration. Please increase the storage in your Elasticsearch cluster."}
{"severity":"INFO","time":"2021-04-19T19:33:40.861Z","correlation_id":"5bd7696d8aac577711d6f7264b01e4b7","message":"MigrationWorker: migration[TestMigration] updating with halted: true"}

Note: to reset the migration you will need to unset the halted state

curl --request PUT \
  --url $ES_CLUSTER/gitlab-development-migrations/_doc/20210301150000 \
  --data '{
	"completed": false
	"state": {},
	"started_at": "2021-01-07T14:14:30.291Z"
}'

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Terri Chu

Merge request reports