Orphaned Elasticsearch migrations for disabled integrations

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue
• Close this issue

Summary

Pending Elastic migrations are left behind for disabled Elasticsearch integrations

https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch.html#disable-advanced-search

Steps to reproduce

1. Configure Elasticsearch integrations and enable indexing.
2. Remove or shutdown Elasticsearch instance/cluster
3. Follow Disable Advanced Search documentation
4. "Delete the existing indices" step cannot be completed because the Elasticsearch node is no longer online, and you'll get Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool errors.
5. Overtime Pending Migrations list will keep growing and customers may start asking questions about those as in this internal ticket

What is the current bug behavior?

There is no way to completely disable Advanced search after the Elasticsearch instance is removed and no longer reachable.

What is the expected correct behavior?

We should implement a --force disable option to reset Advanced search integration state to clear these orphaned pending migrations.

Relevant logs and/or screenshots

root@main-gl-instance:~# sudo gitlab-rake gitlab:elastic:info

Advanced Search
Server version:		unknown
Server distribution:	unknown
Indexing enabled:	no
Search enabled:		no
Pause indexing:		no
File size limit:	1024 KiB

Indexing Queues
Initial queue:		16
Incremental queue:	32

Pending Migrations
ApplyMaxAnalyzedOffset
AddNewDataToIssuesDocuments
MigrateIssuesToSeparateIndex
DeleteIssuesFromOriginalIndex
RemovePermissionsDataFromNotesDocuments
AddPermissionsDataToNotesDocuments
MigrateNotesToSeparateIndex
AddNewDataToMergeRequestsDocuments
MigrateMergeRequestsToSeparateIndex
DeleteMergeRequestsFromOriginalIndex
DeleteNotesFromOriginalIndex
AddUpvotesToIssues
AddUpvotesMappingsToMergeRequests
AddNamespaceAncestryToIssuesMapping
BackfillNamespaceAncestryForIssues
AddNamespaceAncestryIdsToIssuesMapping
RedoBackfillNamespaceAncestryIdsForIssues
DeleteOrphanedCommits
PopulateCommitPermissionsInMainIndex
PauseIndexingForUnsupportedEsVersions
MigrateCommitsToSeparateIndex
DeleteCommitsFromOriginalIndex
An exception occurred during the retrieval of the data: Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.

Indices
  - failed to load indices for gitlab-production
  - failed to load indices for gitlab-production-commits
  - failed to load indices for gitlab-production-issues
  - failed to load indices for gitlab-production-merge_requests
  - failed to load indices for gitlab-production-notes
root@main-gl-instance:~# sudo gitlab-rake gitlab:elastic:delete_index
rake aborted!
Elasticsearch::Transport::Transport::Error: Cannot get new connection from pool.
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/helper.rb:189:in `alias_exists?'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/helper.rb:303:in `target_index_name'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/elastic/helper.rb:177:in `delete_index'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:122:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:elastic:delete_index
(See full trace by running task with --trace)
root@main-gl-instance:~# sudo gitlab-rake gitlab:elastic:disable_search_with_elasticsearch
Setting `elasticsearch_search` was already disabled.

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		Ubuntu 20.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.5p203
Gem Version:	3.1.4
Bundler Version:2.3.15
Rake Version:	13.0.6
Redis Version:	6.2.7
Sidekiq Version:6.4.0
Go Version:	unknown

GitLab information
Version:	15.2.3-ee
Revision:	3317b9ff0d3
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.10
URL:		https://main.gitlab.egrechishkina.com
HTTP Clone URL:	https://main.gitlab.egrechishkina.com/some-group/some-project.git
SSH Clone URL:	git@main.gitlab.egrechishkina.com:some-group/some-project.git
Elasticsearch:	no
Geo:		yes
Geo node:	Primary
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	14.9.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.9.0 ? ... OK (14.9.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
51/2 ... yes
51/3 ... yes
51/4 ... yes
51/5 ... yes
51/6 ... yes
51/7 ... yes
51/8 ... yes
51/9 ... yes
51/10 ... yes
51/11 ... yes
52/12 ... yes
52/13 ... yes
53/14 ... yes
53/15 ... yes
53/16 ... yes
54/17 ... yes
54/18 ... yes
54/19 ... yes
54/20 ... yes
54/21 ... yes
54/22 ... yes
54/23 ... yes
54/24 ... yes
54/25 ... yes
54/26 ... yes
55/27 ... yes
55/28 ... yes
55/29 ... yes
55/30 ... yes
56/31 ... yes
56/32 ... yes
56/33 ... yes
56/34 ... yes
56/35 ... yes
57/36 ... yes
57/37 ... yes
57/38 ... yes
57/39 ... yes
58/40 ... yes
58/41 ... yes
58/42 ... yes
66/43 ... yes
70/45 ... yes
70/46 ... yes
69/47 ... yes
69/48 ... yes
58/52 ... yes
58/53 ... yes
63/54 ... yes
66/55 ... yes
66/56 ... yes
66/57 ... yes
66/58 ... yes
66/59 ... yes
65/72 ... yes
66/77 ... yes
66/78 ... yes
69/81 ... yes
66/85 ... yes
66/86 ... yes
66/89 ... yes
66/90 ... yes
66/92 ... yes
66/97 ... yes
69/100 ... yes
66/101 ... yes
66/102 ... yes
66/103 ... yes
66/104 ... yes
66/107 ... yes
66/110 ... yes
69/111 ... yes
94/112 ... yes
66/113 ... yes
69/114 ... yes
91/115 ... yes
66/117 ... yes
66/118 ... yes
66/119 ... yes
66/121 ... yes
66/122 ... yes
66/124 ... yes
66/125 ... yes
66/126 ... yes
66/128 ... yes
66/129 ... yes
66/130 ... yes
66/131 ... yes
69/132 ... yes
66/133 ... yes
66/137 ... yes
91/138 ... yes
69/140 ... yes
69/141 ... yes
69/142 ... yes
66/143 ... yes
100/144 ... yes
102/145 ... yes
69/146 ... yes
66/147 ... yes
66/148 ... yes
66/149 ... yes
66/152 ... yes
69/153 ... yes
87/154 ... yes
66/155 ... yes
66/156 ... yes
66/157 ... yes
66/159 ... yes
66/160 ... yes
66/161 ... yes
106/162 ... yes
69/163 ... yes
60/164 ... yes
108/165 ... yes
66/166 ... yes
112/167 ... yes
111/168 ... yes
66/169 ... yes
66/171 ... yes
66/172 ... yes
66/173 ... yes
66/174 ... yes
66/175 ... yes
66/176 ... yes
66/177 ... yes
91/180 ... yes
120/184 ... yes
1/185 ... yes
1/186 ... yes
1/187 ... yes
66/189 ... yes
1/190 ... yes
1/191 ... yes
66/192 ... yes
130/193 ... yes
1/194 ... yes
272/195 ... yes
275/197 ... yes
1/198 ... yes
54/199 ... yes
54/200 ... yes
54/201 ... yes
272/202 ... yes
272/203 ... yes
1/204 ... yes
54/205 ... yes
272/206 ... yes
62/207 ... yes
54/208 ... yes
1/209 ... yes
54/210 ... yes
54/211 ... yes
54/212 ... yes
63/213 ... yes
59/214 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 59
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking Geo ...
GitLab Geo is available ...
GitLab Geo is enabled ... yes
This machine's Geo node name matches a database record ... yes, found a primary node named "main.gitlab.egrechishkina.com"
HTTP/HTTPS repository cloning is enabled ... yes
Machine clock is synchronized ... yes
Git user has default SSH configuration? ... yes
OpenSSH configured to use AuthorizedKeysCommand ... yes
GitLab configured to disable writing to authorized_keys file ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking Geo ... Finished
Checking GitLab subtasks ... Finished

Possible fixes