Skip to content

Move gitlab:elastic rake tasks to service

Terri Chu requested to merge 421298-fix-move-all-rake-tasks-to-service into master

What does this MR do and why?

Related to #421298 (closed)

Part 2 of 3 MRs to change the gitlab:elastic:*** rake tasks into a service class and async worker

Description Merge Request
Change gitlab:elastic rake tasks to use a custom stdout logger !153855 (merged)
⭐ Move gitlab:elastic rake code into RakeTaskExecutorService !154484 (merged)
Fully move gitlab:elastic:index to an async worker !154610 (merged)

What this does

  • Change RakeTaskExecutorService to use provided logger for all output
  • Move all calls from gitlab:elastic:* rake into the RakeTaskExecutorService
  • Refactor elastic rake specs to check the service is called
  • Move all elastic rake specs to RakeTaskExecutorService specs

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

There should be no changes, the rake task code is just moving locations.

How to set up and validate locally

  1. ensure you have gdk setup for elasticsearch and that advanced search is configured
  2. run each rake task that was moved to the service
gitlab:elastic:index_group_entities 
➜ bundle exec rake gitlab:elastic:index_group_entities
Enqueuing Group level entities…
Indexing epics...
Indexing epics... done
Indexing group wikis...
Indexing group wikis... done
gitlab:elastic:index_group_wikis 
➜ bundle exec rake gitlab:elastic:index_group_wikis
Indexing group wikis...
Indexing group wikis... done
gitlab:elastic:delete_index 
➜ bundle exec rake gitlab:elastic:delete_index
Index/alias 'gitlab-development' has been deleted
Index 'gitlab-development-issues-20240529-0118' with alias 'gitlab-development-issues' has been deleted
Index 'gitlab-development-notes-20240529-0118' with alias 'gitlab-development-notes' has been deleted
Index 'gitlab-development-merge_requests-20240529-0118' with alias 'gitlab-development-merge_requests' has been deleted
Index 'gitlab-development-commits-20240529-0118' with alias 'gitlab-development-commits' has been deleted
Index 'gitlab-development-epics-20240529-0118' with alias 'gitlab-development-epics' has been deleted
Index 'gitlab-development-users-20240529-0118' with alias 'gitlab-development-users' has been deleted
Index 'gitlab-development-wikis-20240529-0118' with alias 'gitlab-development-wikis' has been deleted
Index 'gitlab-development-projects-20240529-0118' with alias 'gitlab-development-projects' has been deleted
Index/alias 'gitlab-development-migrations' has been deleted
gitlab:elastic:clear_index_status 
➜ bundle exec rake gitlab:elastic:clear_index_status
Index status has been reset
gitlab:elastic:create_empty_index 
➜ bundle exec rake gitlab:elastic:create_empty_index
Index 'gitlab-development-issues-20240603-1807' has been created.
Alias 'gitlab-development-issues' -> 'gitlab-development-issues-20240603-1807' has been created.
Index 'gitlab-development-notes-20240603-1807' has been created.
Alias 'gitlab-development-notes' -> 'gitlab-development-notes-20240603-1807' has been created.
Index 'gitlab-development-merge_requests-20240603-1807' has been created.
Alias 'gitlab-development-merge_requests' -> 'gitlab-development-merge_requests-20240603-1807' has been created.
Index 'gitlab-development-commits-20240603-1807' has been created.
Alias 'gitlab-development-commits' -> 'gitlab-development-commits-20240603-1807' has been created.
Index 'gitlab-development-epics-20240603-1807' has been created.
Alias 'gitlab-development-epics' -> 'gitlab-development-epics-20240603-1807' has been created.
Index 'gitlab-development-users-20240603-1807' has been created.
Alias 'gitlab-development-users' -> 'gitlab-development-users-20240603-1807' has been created.
Index 'gitlab-development-wikis-20240603-1807' has been created.
Alias 'gitlab-development-wikis' -> 'gitlab-development-wikis-20240603-1807' has been created.
Index 'gitlab-development-projects-20240603-1807' has been created.
Alias 'gitlab-development-projects' -> 'gitlab-development-projects-20240603-1807' has been created.
Index '{"gitlab-development-20240603-1807"=>"gitlab-development"}' has been created.
Alias 'gitlab-development' → '{"gitlab-development-20240603-1807"=>"gitlab-development"}' has been created
gitlab:elastic:recreate_index 
➜ bundle exec rake gitlab:elastic:recreate_index
Index/alias 'gitlab-development' has been deleted
Index 'gitlab-development-issues-20240603-1807' with alias 'gitlab-development-issues' has been deleted
Index 'gitlab-development-notes-20240603-1807' with alias 'gitlab-development-notes' has been deleted
Index 'gitlab-development-merge_requests-20240603-1807' with alias 'gitlab-development-merge_requests' has been deleted
Index 'gitlab-development-commits-20240603-1807' with alias 'gitlab-development-commits' has been deleted
Index 'gitlab-development-epics-20240603-1807' with alias 'gitlab-development-epics' has been deleted
Index 'gitlab-development-users-20240603-1807' with alias 'gitlab-development-users' has been deleted
Index 'gitlab-development-wikis-20240603-1807' with alias 'gitlab-development-wikis' has been deleted
Index 'gitlab-development-projects-20240603-1807' with alias 'gitlab-development-projects' has been deleted
Index/alias 'gitlab-development-migrations' has been deleted
ohIndex 'gitlab-development-issues-20240603-1808' has been created.
Alias 'gitlab-development-issues' -> 'gitlab-development-issues-20240603-1808' has been created.
Index 'gitlab-development-notes-20240603-1808' has been created.
Alias 'gitlab-development-notes' -> 'gitlab-development-notes-20240603-1808' has been created.
Index 'gitlab-development-merge_requests-20240603-1808' has been created.
Alias 'gitlab-development-merge_requests' -> 'gitlab-development-merge_requests-20240603-1808' has been created.
Index 'gitlab-development-commits-20240603-1808' has been created.
Alias 'gitlab-development-commits' -> 'gitlab-development-commits-20240603-1808' has been created.
Index 'gitlab-development-epics-20240603-1808' has been created.
Alias 'gitlab-development-epics' -> 'gitlab-development-epics-20240603-1808' has been created.
Index 'gitlab-development-users-20240603-1808' has been created.
Alias 'gitlab-development-users' -> 'gitlab-development-users-20240603-1808' has been created.
Index 'gitlab-development-wikis-20240603-1808' has been created.
Alias 'gitlab-development-wikis' -> 'gitlab-development-wikis-20240603-1808' has been created.
Index 'gitlab-development-projects-20240603-1808' has been created.
Alias 'gitlab-development-projects' -> 'gitlab-development-projects-20240603-1808' has been created.
Index '{"gitlab-development-20240603-1808"=>"gitlab-development"}' has been created.
Alias 'gitlab-development' → '{"gitlab-development-20240603-1808"=>"gitlab-development"}' has been created
gitlab:elastic:projects_not_indexed 
➜ bundle exec rake gitlab:elastic:projects_not_indexed
Project 'a-zoekt-indexed-group/a-small-project' (ID: 39) isn't indexed.
Project 'namespace5/project-5' (ID: 55) isn't indexed.
Project 'namespace6/project-6' (ID: 56) isn't indexed.
Project 'flightjs/subgroup-b/project-a' (ID: 41) isn't indexed.
Project 'a-zoekt-indexed-group/gitlab-hq' (ID: 37) isn't indexed.
Project 'zoekt-indexing/i-have-many-branches' (ID: 27) isn't indexed.
Project 'root/a-user-project' (ID: 40) isn't indexed.
Project 'toolbox/gitlab-smoke-tests' (ID: 1) isn't indexed.
Project 'a-zoekt-indexed-group/Flight' (ID: 46) isn't indexed.
Project 'a-zoekt-indexed-group/archived-project' (ID: 48) isn't indexed.
Project 'toolbox/add-me-to-the-index' (ID: 31) isn't indexed.
Project 'gitlab-org/i-am-already-indexed' (ID: 32) isn't indexed.
Project 'submodule-group-test/project-with-submodule' (ID: 29) isn't indexed.
Project 'permissions-test/project-with-a-binary-file' (ID: 28) isn't indexed.
Project 'gitlab-org/gitlab-test' (ID: 2) isn't indexed.
Project 'gnuwget/Wget2' (ID: 4) isn't indexed.
Project 'gitlab-org/gitlab-shell' (ID: 3) isn't indexed.
Project 'permissions-test/public-project-repository-disabled' (ID: 26) isn't indexed.
Project 'a-not-indexed-group/a-not-indexed-project' (ID: 36) isn't indexed.
Project 'namespace7/project-7' (ID: 57) isn't indexed.
Project 'jashkenas/Underscore' (ID: 6) isn't indexed.
Project 'kai/gitlab-shell' (ID: 11) isn't indexed.
Project 'Commit451/a-256-sha-project' (ID: 43) isn't indexed.
Project 'francis.howe/gitlab-shell' (ID: 9) isn't indexed.
Project 'elke/gitlab-shell' (ID: 10) isn't indexed.
Project 'zoekt-indexing/airflow-repo-template' (ID: 44) isn't indexed.
Project 'Commit451/test-no-repo' (ID: 45) isn't indexed.
Project 'a-zoekt-indexed-group/a-project-with-empty-repo' (ID: 47) isn't indexed.
Project 'submodule-group-test/i-am-a-submodule' (ID: 30) isn't indexed.
Project 'twitter/twitter-subgroup/Typeahead.Js' (ID: 8) isn't indexed.
Project 'Commit451/lab-coat' (ID: 5) isn't indexed.
Project 'flightjs/Flight' (ID: 7) isn't indexed.
Project 'testing-elasticsearch-indexer/airflow-repo-template' (ID: 42) isn't indexed.
Project 'namespace1/project-1' (ID: 51) isn't indexed.
Project 'namespace2/project-2' (ID: 52) isn't indexed.
Project 'namespace3/project-3' (ID: 53) isn't indexed.
Project 'Commit451/another-group/regex-test' (ID: 50) isn't indexed.
Project 'namespace4/project-4' (ID: 54) isn't indexed.
Project 'a-zoekt-indexed-group/i-have-many-branches' (ID: 49) isn't indexed.
Project 'cedric/gitlab-shell' (ID: 12) isn't indexed.
Project 'hyo.ward/gitlab-shell' (ID: 13) isn't indexed.
Project 'christel/gitlab-shell' (ID: 14) isn't indexed.
Project 'christel_littel/gitlab-shell' (ID: 15) isn't indexed.
Project 'davina.bartell/gitlab-shell' (ID: 16) isn't indexed.
Project 'Commit451/another-group/another-subgroup/my-transferred-project' (ID: 58) isn't indexed.
Project 'Commit451/another-group/another-subgroup/transfer-me' (ID: 59) isn't indexed.
Project 'gnuwget/test-transfer' (ID: 60) isn't indexed.
Project 'not-indexed-group/public-project' (ID: 19) isn't indexed.
Project 'permissions-test/private-project' (ID: 21) isn't indexed.
Project 'permissions-test/internal-project' (ID: 20) isn't indexed.
Project 'permissions-test/public-subgroup/public-project' (ID: 22) isn't indexed.
Project 'permissions-test/internal-subgroup/private-project' (ID: 24) isn't indexed.
Project 'permissions-test/internal-subgroup/internal-project' (ID: 23) isn't indexed.
Project 'permissions-test/private-subgroup/private-project' (ID: 25) isn't indexed.
Project 'frederick/gitlab-shell' (ID: 17) isn't indexed.
Project 'danial/gitlab-shell' (ID: 18) isn't indexed.
56 out of 56 non-indexed projects shown.
gitlab:elastic:reindex_cluster 
➜ bundle exec rake gitlab:elastic:reindex_cluster
Reindexing job was successfully scheduled
gitlab:elastic:info 
➜ bundle exec rake gitlab:elastic:info

Advanced Search
Server version:			8.11.1
Server distribution:		elasticsearch
Indexing enabled:		yes
Search enabled:			no
Requeue Indexing workers:	yes
Pause indexing:			yes
Indexing restrictions enabled:	no
File size limit:		1024 KiB
Indexing number of shards:	2
Max code indexing concurrency:	30

Indexing Queues
Initial queue:			995
Incremental queue:		40

Indices
- gitlab-development-20240603-1812:
	document_count: 2673
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-commits-20240603-1812:
	document_count: 26243
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-epics-20240603-1812:
	document_count: 0
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-issues-20240603-1812:
	document_count: 0
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-merge_requests-20240603-1812:
	document_count: 0
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-notes-20240603-1812:
	document_count: 0
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-projects-20240603-1812:
	document_count: 0
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-users-20240603-1812:
	document_count: 0
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-wikis-20240603-1812:
	document_count: 8
	number_of_shards: 5
	number_of_replicas: 1
- gitlab-development-work_items-20240423-1216:
	document_count: 0
	number_of_shards: 5
	number_of_replicas: 1
  1. run each rake task where the executor service was updated to use logger vs puts
gitlab:elastic:estimate_cluster_size 
➜ bundle exec rake gitlab:elastic:estimate_cluster_size
This GitLab instance combined repository and wiki size is 803.4 MiB.
By our estimates, your cluster size should be at least 401.7 MiB.
Please note that it is possible to index only selected namespaces/projects by using Advanced search indexing restrictions.
gitlab:elastic:mark_reindex_failed 
➜ bundle exec rake gitlab:elastic:mark_reindex_failed
Marked the current reindexing job as failed.

➜ bundle exec rake gitlab:elastic:mark_reindex_failed
Did not find the current running reindexing job.
gitlab:elastic:list_pending_migrations 
➜ bundle exec rake gitlab:elastic:list_pending_migrations
There are no pending migrations.

➜ bundle exec rake gitlab:elastic:list_pending_migrations
Pending Migrations
BackfillArchivedOnWorkItems [Obsolete]
IndexAllProjects
gitlab:elastic:enable_search_with_elasticsearch 
➜ bundle exec rake gitlab:elastic:enable_search_with_elasticsearch
Setting `elasticsearch_search` has been enabled.
gitlab:elastic:index_projects_status 
➜ bundle exec rake gitlab:elastic:index_projects_status
Indexing is 100.00% complete (56/56 projects)
gitlab:elastic:index_projects 
➜ bundle exec rake gitlab:elastic:index_projects
Enqueuing projects…
. ✔ (56)
gitlab:elastic:estimate_shard_sizes 
➜ bundle exec rake gitlab:elastic:estimate_shard_sizes
Using approximate counts to estimate shard counts for data indexed from database. This does not include repository data. For single-node cluster recommendations, see http://gdk.test:3000/help/integration/advanced_search/elasticsearch#number-of-elasticsearch-shards.
The approximate document counts, recommended shard size, and replica size for each index are:
- gitlab-development-issues:
  document count: 671
  recommended shards: 5
  recommended replicas: 1

- gitlab-development-notes:
  document count: 163
  recommended shards: 5
  recommended replicas: 1

- gitlab-development-merge_requests:
  document count: 107
  recommended shards: 5
  recommended replicas: 1

- gitlab-development-epics:
  document count: 40
  recommended shards: 5
  recommended replicas: 1

- gitlab-development-users:
  document count: 61
  recommended shards: 5
  recommended replicas: 1

- gitlab-development-projects:
  document count: 56
  recommended shards: 5
  recommended replicas: 1

Please note that it is possible to index only selected namespaces/projects by using Advanced search indexing restrictions. This estimate does not take into account indexing restrictions.
Edited by Terri Chu

Merge request reports