Background migrations stuck as "active" after upgrade from 15.3.5 to 15.4.6
### Summary
After upgrading from 15.3.5 to 15.4.6, 16 background migrations are stuck at an "active" state with 0% progress. I went through the docs multiple times but could not figure out how to proceed.
What is the procedure to manually trigger these migrations? I started to search for these `job_class_name`s and found e.g. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91360 but I am confused how to resolve it.
### Steps to reproduce
All I did was upgrading from 15.3.5 to 15.4.6
### What is the current *bug* behavior?
A total of 16 background migrations are pending in "active" state with 0% progress for several days
### What is the expected *correct* behavior?
Background migrations should complete after upgrade.
### Relevant logs and/or screenshots
Here is the db query which shows the 16 migrations:
```
gitlabhq_production=# select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3;
job_class_name | table_name | column_name | job_arguments
------------------------------------------------------+---------------------------+--------------+---------------
UpdateDelayedProjectRemovalToNullForUserNamespaces | namespace_settings | namespace_id | []
BackfillImportedIssueSearchData | issues | id | []
BackfillProjectImportLevel | namespaces | id | []
MigrateSharedVulnerabilityScanners | vulnerability_occurrences | id | []
RemoveSelfManagedWikiNotes | notes | id | []
RemoveBackfilledJobArtifactsExpireAt | ci_job_artifacts | id | []
BackfillProjectNamespaceOnIssues | issues | id | []
DestroyInvalidGroupMembers | members | id | []
RenameTaskSystemNoteToChecklistItem | system_note_metadata | id | []
BackfillWorkItemTypeIdForIssues | issues | id | [0, 1]
BackfillWorkItemTypeIdForIssues | issues | id | [1, 2]
BackfillWorkItemTypeIdForIssues | issues | id | [2, 3]
BackfillWorkItemTypeIdForIssues | issues | id | [3, 4]
BackfillWorkItemTypeIdForIssues | issues | id | [4, 5]
DestroyInvalidProjectMembers | members | id | []
PopulateOperationVisibilityPermissionsFromOperations | project_features | id | []
(16 rows)
```
Here are a few checks via rails:
```
# gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
WARNING: Active Record does not support composite primary key.
security_findings has composite primary key. Composite primary key is ignored.
WARNING: Active Record does not support composite primary key.
security_findings has composite primary key. Composite primary key is ignored.
0
# gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.queued.count'
WARNING: Active Record does not support composite primary key.
security_findings has composite primary key. Composite primary key is ignored.
WARNING: Active Record does not support composite primary key.
security_findings has composite primary key. Composite primary key is ignored.
16
# gitlab-rails runner -e production 'puts Gitlab::Database::BackgroundMigration::BatchedMigration.with_status(:failed).count'
WARNING: Active Record does not support composite primary key.
security_findings has composite primary key. Composite primary key is ignored.
WARNING: Active Record does not support composite primary key.
security_findings has composite primary key. Composite primary key is ignored.
0
```
### Output of checks
<!-- If you are reporting a bug on GitLab.com, uncomment below -->
<!-- This bug happens on GitLab.com -->
<!-- /label ~"reproduced on GitLab.com" -->
#### Results of GitLab environment info
<!-- Input any relevant GitLab environment information if needed. -->
<details>
<summary>Expand for output related to GitLab environment info</summary>
<pre>
# gitlab-rake gitlab:env:info
System information
System: Ubuntu 20.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.7.6p219
Gem Version: 3.1.6
Bundler Version:2.3.15
Rake Version: 13.0.6
Redis Version: 6.2.7
Sidekiq Version:6.4.2
Go Version: unknown
GitLab information
Version: 15.4.6-ee
Revision: 4fc991ae59a
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 13.8
URL: https://git.ecap.work
HTTP Clone URL: https://git.ecap.work/some-group/some-project.git
SSH Clone URL: git@git.ecap.work:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 14.10.0
Repository storage paths:
- default: /raid/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
</pre>
</details>
#### Results of GitLab application Check
<!-- Input any relevant GitLab application check information if needed. -->
<details>
<summary>Expand for output related to the GitLab application check</summary>
<pre>
# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.10.0 ? ... OK (14.10.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/32
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
3/2 ... yes
4/7 ... yes
5/11 ... yes
4/12 ... yes
4/13 ... yes
3/14 ... yes
5/16 ... yes
5/17 ... yes
5/18 ... yes
5/19 ... yes
5/23 ... yes
5/24 ... yes
3/25 ... yes
5/28 ... yes
3/30 ... yes
4/31 ... yes
12/33 ... yes
10/34 ... yes
2/35 ... yes
15/37 ... yes
12/38 ... yes
5/42 ... yes
21/43 ... yes
23/44 ... yes
12/46 ... yes
12/47 ... yes
26/48 ... yes
14/49 ... yes
14/51 ... yes
29/53 ... yes
29/54 ... yes
29/55 ... yes
5/56 ... yes
14/57 ... yes
14/58 ... yes
34/60 ... yes
15/61 ... yes
35/62 ... yes
29/63 ... yes
29/64 ... yes
15/65 ... yes
15/66 ... yes
35/67 ... yes
38/68 ... yes
4/69 ... yes
12/71 ... yes
39/72 ... yes
39/73 ... yes
14/75 ... yes
5/76 ... yes
5/77 ... yes
40/78 ... yes
5/79 ... yes
40/80 ... yes
32/82 ... yes
35/83 ... yes
5/84 ... yes
4/85 ... yes
14/86 ... yes
48/87 ... yes
5/88 ... yes
39/89 ... yes
24/90 ... yes
14/91 ... yes
4/92 ... yes
14/93 ... yes
48/94 ... yes
35/95 ... yes
51/96 ... yes
35/97 ... yes
4/98 ... yes
14/99 ... yes
14/100 ... yes
5/104 ... yes
4/105 ... yes
14/106 ... yes
30/107 ... yes
65/108 ... yes
14/109 ... yes
60/110 ... yes
2/111 ... yes
64/112 ... yes
65/113 ... yes
38/114 ... yes
4/115 ... yes
4/116 ... yes
54/117 ... yes
50/118 ... yes
30/119 ... yes
4/120 ... yes
51/121 ... yes
40/122 ... yes
30/123 ... yes
54/124 ... yes
71/125 ... yes
54/126 ... yes
40/127 ... yes
75/128 ... yes
40/129 ... yes
4/130 ... yes
64/132 ... yes
33/133 ... yes
64/136 ... yes
64/137 ... yes
80/139 ... yes
80/140 ... yes
58/141 ... yes
40/142 ... yes
4/143 ... yes
33/144 ... yes
80/145 ... yes
61/146 ... yes
88/147 ... yes
90/149 ... yes
90/150 ... yes
83/151 ... yes
83/152 ... yes
83/153 ... yes
83/154 ... yes
83/155 ... yes
83/156 ... yes
83/157 ... yes
83/158 ... yes
83/159 ... yes
83/160 ... yes
83/161 ... yes
83/162 ... yes
83/163 ... yes
83/164 ... yes
83/165 ... yes
83/166 ... yes
83/167 ... yes
83/168 ... yes
83/169 ... yes
95/170 ... yes
96/171 ... yes
84/172 ... yes
98/173 ... yes
85/174 ... yes
85/175 ... yes
85/176 ... yes
100/177 ... yes
103/178 ... yes
104/179 ... yes
105/180 ... yes
86/181 ... yes
86/182 ... yes
107/183 ... yes
107/184 ... yes
88/185 ... yes
88/186 ... yes
88/187 ... yes
88/188 ... yes
2/189 ... yes
89/190 ... yes
89/191 ... yes
89/192 ... yes
89/193 ... yes
89/194 ... yes
89/195 ... yes
89/196 ... yes
89/197 ... yes
89/198 ... yes
89/199 ... yes
87/200 ... yes
40/201 ... yes
4/202 ... yes
40/204 ... yes
54/205 ... yes
105/206 ... yes
60/207 ... yes
111/208 ... yes
104/209 ... yes
123/210 ... yes
125/211 ... yes
63/212 ... yes
126/213 ... yes
126/215 ... yes
85/216 ... yes
85/218 ... yes
58/220 ... yes
23/221 ... yes
24/223 ... yes
24/224 ... yes
51/225 ... yes
54/226 ... yes
109/227 ... yes
85/228 ... yes
72/229 ... yes
111/230 ... yes
33/232 ... yes
54/233 ... yes
132/234 ... yes
111/236 ... yes
71/237 ... yes
51/239 ... yes
132/242 ... yes
71/244 ... yes
110/245 ... yes
58/246 ... yes
58/247 ... yes
83/248 ... yes
38/250 ... yes
129/251 ... yes
129/252 ... yes
162/254 ... yes
162/255 ... yes
165/256 ... yes
170/257 ... yes
162/258 ... yes
129/259 ... yes
175/260 ... yes
83/261 ... yes
96/262 ... yes
170/263 ... yes
90/264 ... yes
83/265 ... yes
129/266 ... yes
132/267 ... yes
406/268 ... yes
126/269 ... yes
2/270 ... yes
412/272 ... yes
129/273 ... yes
415/274 ... yes
170/275 ... yes
418/276 ... yes
421/277 ... yes
418/278 ... yes
165/279 ... yes
24/280 ... yes
30/281 ... yes
48/282 ... yes
48/284 ... yes
48/286 ... yes
415/287 ... yes
415/288 ... yes
415/289 ... yes
175/290 ... yes
178/291 ... yes
58/292 ... yes
439/293 ... yes
439/294 ... yes
162/295 ... yes
412/296 ... yes
58/297 ... yes
132/298 ... yes
138/301 ... yes
170/303 ... yes
43/304 ... yes
101/305 ... yes
72/306 ... yes
412/307 ... yes
439/308 ... yes
406/309 ... yes
406/310 ... yes
406/311 ... yes
465/312 ... yes
465/313 ... yes
468/314 ... yes
175/315 ... yes
175/316 ... yes
175/317 ... yes
162/318 ... yes
162/319 ... yes
439/320 ... yes
123/321 ... yes
439/322 ... yes
60/324 ... yes
24/325 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.6)
Git user has default SSH configuration? ... yes
Active users: ... 118
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
</pre>
</details>
### Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem. -->
issue