Failing DB Migration: BatchedMigrationRunner::FailedToFinalize deployable_id_convert_to_bigint

When I tried updating to GitLab CE 14.2.3, it failed to start because the background migrations hadn't run yet:

Output of gitlab-ctl reconfigure log

[2021-09-06T14:18:58+02:00] ERROR: Running exception handlers
[2021-09-06T14:18:58+02:00] ERROR: Exception handlers complete
[2021-09-06T14:18:58+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2021-09-06T14:18:58+02:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2021-09-06T14:18:58+02:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210906-67-15ov5ue" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'failed':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"deployments", :column_name=>"id", :job_arguments=>[["deployable_id"], ["deployable_id_convert_to_bigint"]]}
Finalize it manualy by running

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]']


For more information, check the documentation

https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished


/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in ensure_batched_background_migration_is_finished' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210802131812_finalize_convert_deployments_bigint.rb:12:in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'failed':	{:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"deployments", :column_name=>"id", :job_arguments=>[["deployable_id"], ["deployable_id_convert_to_bigint"]]}
Finalize it manualy by running

sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]']


For more information, check the documentation

https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished


/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in ensure_batched_background_migration_is_finished' /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210802131812_finalize_convert_deployments_bigint.rb:12:in up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210802131812 FinalizeConvertDeploymentsBigint: migrating =================
STDERR:
---- End output of "bash"  "/tmp/chef-script20210906-67-15ov5ue" ----
Ran "bash"  "/tmp/chef-script20210906-67-15ov5ue" returned 1

So I ran the command the error suggested and it constantly failed. I downgraded to GitLab CE 14.1.5 and still it is failing with the following output:

Output of background migrations trigger

root@Neon:~# docker exec -it gitlab gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,deployments,id,'[["deployable_id"]\, ["deployable_id_convert_to_bigint"]]'] --trace
** Invoke gitlab:background_migrations:finalize (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab:background_migrations:finalize
rake aborted!
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize: Gitlab::Database::BackgroundMigration::BatchedMigrationRunner::FailedToFinalize
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:74:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration_runner.rb:10:in `finalize'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:13:in `block (3 levels) in '
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:279:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:279:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:279:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.3/exe/rake:27:in `'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:476:in `exec'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/cli.rb:24:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:46:in `block in '
/opt/gitlab/embedded/lib/ruby/site_ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/exe/bundle:34:in `'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Tasks: TOP => gitlab:background_migrations:finalize

What can I do to fix this so that I can upgrade to 14.2? For now the GitLab instance is up and running on 14.1.5.

Note that I am running GitLab in a docker container: gitlab/gitlab-ce:14.1.5-ce.0

Results of GitLab environment info

Expand for output related to GitLab environment info

root@Neon:~# docker exec -it gitlab gitlab-rake gitlab:env:info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.7.2p137
Gem Version:    3.1.4
Bundler Version:2.1.4
Rake Version:   13.0.3
Redis Version:  6.0.14
Git Version:    2.32.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        14.1.5
Revision:       2277a91d2e0
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://
HTTP Clone URL: https:///some-group/some-project.git
SSH Clone URL:  git@:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2

GitLab Shell
Version:        13.19.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

root@Neon:~# docker exec -it gitlab gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.19.1 ? ... OK (13.19.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
3/2 ... yes
3/3 ... yes
4/4 ... yes
4/5 ... yes
4/6 ... yes
4/7 ... yes
4/8 ... yes
4/9 ... yes
4/10 ... yes
4/11 ... yes
4/12 ... yes
4/13 ... yes
5/14 ... yes
7/15 ... yes
7/16 ... yes
7/17 ... yes
2/18 ... yes
8/19 ... yes
8/20 ... yes
8/21 ... yes
8/22 ... yes
8/23 ... yes
8/24 ... yes
8/25 ... yes
8/26 ... yes
8/27 ... yes
8/28 ... yes
8/29 ... yes
8/30 ... yes
8/31 ... yes
8/32 ... yes
8/33 ... yes
8/34 ... yes
8/35 ... yes
8/36 ... yes
8/37 ... yes
8/38 ... yes
8/39 ... yes
8/40 ... yes
8/41 ... yes
8/42 ... yes
8/43 ... yes
8/44 ... yes
8/45 ... yes
2/46 ... yes
2/47 ... yes
2/48 ... yes
10/49 ... yes
2/50 ... yes
11/51 ... yes
14/52 ... yes
14/53 ... yes
2/54 ... yes
10/55 ... yes
2/56 ... yes
11/57 ... yes
11/58 ... yes
2/59 ... yes
11/60 ... yes
17/62 ... yes
17/63 ... yes
18/64 ... yes
4/65 ... yes
4/66 ... yes
17/67 ... yes
19/68 ... yes
52/69 ... yes
19/70 ... yes
19/71 ... yes
19/72 ... yes
19/73 ... yes
38/74 ... yes
20/76 ... yes
19/77 ... yes
17/78 ... yes
17/79 ... yes
38/80 ... yes
38/81 ... yes
2/82 ... yes
2/83 ... yes
21/84 ... yes
5/85 ... yes
51/86 ... yes
2/87 ... yes
10/89 ... yes
10/90 ... yes
2/91 ... yes
2/92 ... yes
5/93 ... yes
17/94 ... yes
17/95 ... yes
17/96 ... yes
17/97 ... yes
17/98 ... yes
18/99 ... yes
38/100 ... yes
29/101 ... yes
2/102 ... yes
10/105 ... yes
52/106 ... yes
33/107 ... yes
2/108 ... yes
18/109 ... yes
52/110 ... yes
2/111 ... yes
42/112 ... yes
42/113 ... yes
52/114 ... yes
52/115 ... yes
2/116 ... yes
34/117 ... yes
17/118 ... yes
17/119 ... yes
36/120 ... yes
36/121 ... yes
36/122 ... yes
36/123 ... yes
38/124 ... yes
19/125 ... yes
19/126 ... yes
41/127 ... yes
38/128 ... yes
41/129 ... yes
2/130 ... yes
2/131 ... yes
41/132 ... yes
41/133 ... yes
41/135 ... yes
42/136 ... yes
42/137 ... yes
42/138 ... yes
38/139 ... yes
38/140 ... yes
38/141 ... yes
38/142 ... yes
38/143 ... yes
38/144 ... yes
38/145 ... yes
38/146 ... yes
38/147 ... yes
44/148 ... yes
38/149 ... yes
2/150 ... yes
42/151 ... yes
42/152 ... yes
42/153 ... yes
2/154 ... yes
38/155 ... yes
38/156 ... yes
38/157 ... yes
38/158 ... yes
2/159 ... yes
38/160 ... yes
38/161 ... yes
38/162 ... yes
38/163 ... yes
38/164 ... yes
38/165 ... yes
46/166 ... yes
49/167 ... yes
49/168 ... yes
49/169 ... yes
2/170 ... yes
38/171 ... yes
2/172 ... yes
36/173 ... yes
2/174 ... yes
46/175 ... yes
53/176 ... yes
53/178 ... yes
53/179 ... yes
53/180 ... yes
53/182 ... yes
53/183 ... yes
53/184 ... yes
53/185 ... yes
2/186 ... yes
38/187 ... yes
53/188 ... yes
49/189 ... yes
49/190 ... yes
49/192 ... yes
2/193 ... yes
2/194 ... yes
38/195 ... yes
62/196 ... yes
38/197 ... yes
38/198 ... yes
38/199 ... yes
2/200 ... yes
2/201 ... yes
63/202 ... yes
53/203 ... yes
38/204 ... yes
64/205 ... yes
2/206 ... yes
2/207 ... yes
2/208 ... yes
38/209 ... yes
2/210 ... yes
2/211 ... yes
38/212 ... yes
38/213 ... yes
2/214 ... yes
38/215 ... yes
38/216 ... yes
53/217 ... yes
53/218 ... yes
2/219 ... yes
2/220 ... yes
2/221 ... yes
2/222 ... yes
38/223 ... yes
66/224 ... yes
38/225 ... yes
2/226 ... yes
2/227 ... yes
2/228 ... yes
2/229 ... yes
67/230 ... yes
67/231 ... yes
2/232 ... yes
38/233 ... yes
38/234 ... yes
2/235 ... yes
2/236 ... yes
66/237 ... yes
66/238 ... yes
2/239 ... yes
38/240 ... yes
2/241 ... yes
2/242 ... yes
2/243 ... yes
69/244 ... yes
2/245 ... yes
66/246 ... yes
70/247 ... yes
66/248 ... yes
71/249 ... yes
2/250 ... yes
2/251 ... yes
2/252 ... yes
38/253 ... yes
2/254 ... yes
38/255 ... yes
2/256 ... yes
38/257 ... yes
38/258 ... yes
77/259 ... yes
2/260 ... yes
38/261 ... yes
38/262 ... yes
2/263 ... yes
77/264 ... yes
42/265 ... yes
77/266 ... yes
77/267 ... yes
66/268 ... yes
66/269 ... yes
2/270 ... yes
2/271 ... yes
2/272 ... yes
2/273 ... yes
2/274 ... yes
92/275 ... yes
92/276 ... yes
2/277 ... yes
2/278 ... yes
2/279 ... yes
67/280 ... yes
2/281 ... yes
2/282 ... yes
2/283 ... yes
66/284 ... yes
2/285 ... yes
92/286 ... yes
2/287 ... yes
38/288 ... yes
92/289 ... yes
99/290 ... yes
38/291 ... yes
67/292 ... yes
2/293 ... yes
2/294 ... yes
67/295 ... yes
92/296 ... yes
119/297 ... yes
2/298 ... yes
2/299 ... yes
2/300 ... yes
2/301 ... yes
2/302 ... yes
115/303 ... yes
66/304 ... yes
118/305 ... yes
2/306 ... yes
119/307 ... yes
70/308 ... yes
70/309 ... yes
115/310 ... yes
2/311 ... yes
2/312 ... yes
2/313 ... yes
114/314 ... yes
66/315 ... yes
115/316 ... yes
114/317 ... yes
122/318 ... yes
122/319 ... yes
2/320 ... yes
122/321 ... yes
122/322 ... yes
122/323 ... yes
2/324 ... yes
38/325 ... yes
38/326 ... yes
122/327 ... yes
2/328 ... yes
92/329 ... yes
124/330 ... yes
77/331 ... yes
2/332 ... yes
77/333 ... yes
77/334 ... yes
2/335 ... yes
122/336 ... yes
38/337 ... yes
42/338 ... yes
122/339 ... yes
67/340 ... yes
122/341 ... yes
126/342 ... yes
122/343 ... yes
128/344 ... yes
128/345 ... yes
2/346 ... yes
129/347 ... yes
122/348 ... yes
38/349 ... yes
2/351 ... yes
132/352 ... yes
132/353 ... yes
132/354 ... yes
132/355 ... yes
38/357 ... yes
122/358 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.32.0)
Git user has default SSH configuration? ... yes
Active users: ... 23
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished