Skip to content

Refactor mark_all_as_completed! to use bulk requests

Terri Chu requested to merge tchu-improve-mark_all_migrations_complete-speed into master

What does this MR do and why?

I've noticed specs that use any :elastic* traits have been slower lately and found during a pair session that each migration is marked as completely individually. There are about 100 Advanced search migrations and this problem is only going worsen as more migrations are added. Obsolete migrations are also loaded

This MR moves to using a bulk request to index all migrations into elastic. There is a limit to how large bulk requests can be, but we can iterate on how obsolete migrations are handled to avoid hitting this.

Screenshots or screen recordings

This change only affects the specs and elastic.rake task when search is setup from scratch or indexes are recreated from scratch

N/A

How to set up and validate locally

Note: setup gdk to use elasticsearch and setup/enable Advanced search

  1. open rails console and load a Elastic helper: helper = Gitlab::Elastic::Helper.default
  2. delete the migrations index: helper.delete_index(index_name: helper.migrations_index_name)
  3. create the migrations index: helper.create_migrations_index
  4. mark all migrations as completed: Elastic::DataMigrationService.mark_all_as_completed!
  5. verify that from curl
curl --request GET \
  --url http://localhost:9200/gitlab-development-migrations/_search \
  --header 'Content-Type: application/json' \
  --data '{
	"query": {
		"bool": {
			"must": [
				{
					"match_all": {}
				}
			],
			"filter": [
				
			]
		}
	},
	"size": 1500
}'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Terri Chu

Merge request reports