Gitlab Backup failing after 14.9.2 upgrade

Summary

Since the upgrade to version 14.9.2 our Gitlab backups are failing. If I restart Gitlab, backup is fine again but after a few days it's failing again.

Steps to reproduce

/usr/bin/gitlab-rake gitlab:backup:create

Example Project

N/A

What is the current bug behavior?

/usr/bin/gitlab-rake gitlab:backup:create

Error:

rake aborted!
Backup::Error: gitaly-backup exit status 1
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:60:in `finish!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/repositories.rb:54:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:113:in `run_create_task'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:77:in `block in create'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:76:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:76:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:58:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:58:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli/exec.rb:23:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:479:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:31:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/cli.rb:25:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/exe/bundle:49:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.2.33/exe/bundle:37:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create

What is the expected correct behavior?

Backup is going fine.

Relevant logs and/or screenshots

See above

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

```
System information
System:         RedHatEnterprise 8.5
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.1.4
Bundler Version:2.2.33
Rake Version:   13.0.6
Redis Version:  6.2.6
Sidekiq Version:6.4.0
Go Version:     unknown

GitLab information
Version:        14.9.2-ee
Revision:       3034418fb31
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://gitlab.test.colruytgroup.com
HTTP Clone URL: https://gitlab.test.colruytgroup.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.test.colruytgroup.com:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.24.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
```

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.24.0 ? ... OK (13.24.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
	User output sanitized. Found 93 users of 100 limit.

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... no
  Try fixing it:
  sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ... 
4/1 ... yes
6/2 ... yes
8/3 ... yes
8/4 ... yes
12/5 ... yes
12/6 ... yes
8/7 ... yes
14/8 ... yes
2/9 ... yes
8/10 ... yes
5/11 ... yes
39/12 ... yes
5/13 ... yes
24/14 ... yes
26/15 ... yes
8/18 ... yes
8/19 ... yes
12/20 ... yes
7/21 ... yes
36/22 ... yes
7/24 ... yes
7/25 ... yes
7/26 ... yes
7/27 ... yes
18/28 ... yes
24/29 ... yes
24/30 ... yes
24/31 ... yes
42/33 ... yes
42/34 ... yes
42/35 ... yes
42/36 ... yes
42/37 ... yes
42/38 ... yes
42/39 ... yes
36/40 ... yes
42/41 ... yes
42/42 ... yes
42/43 ... yes
42/44 ... yes
50/46 ... yes
50/47 ... yes
51/48 ... yes
51/49 ... yes
62/50 ... yes
50/51 ... yes
53/52 ... yes
53/53 ... yes
51/54 ... yes
51/55 ... yes
55/56 ... yes
55/57 ... yes
62/58 ... yes
62/59 ... yes
62/60 ... yes
62/61 ... yes
62/62 ... yes
56/63 ... yes
57/64 ... yes
62/65 ... yes
62/66 ... yes
58/67 ... yes
59/68 ... yes
51/69 ... yes
60/70 ... yes
57/71 ... yes
42/72 ... yes
61/73 ... yes
57/74 ... yes
53/76 ... yes
62/77 ... yes
53/78 ... yes
62/80 ... yes
62/81 ... yes
53/83 ... yes
62/84 ... yes
57/85 ... yes
64/86 ... yes
57/87 ... yes
50/90 ... yes
50/91 ... yes
62/93 ... yes
62/95 ... yes
7/96 ... yes
57/97 ... yes
57/98 ... yes
57/99 ... yes
62/100 ... yes
62/101 ... yes
62/103 ... yes
57/104 ... yes
67/105 ... yes
69/107 ... yes
71/108 ... yes
73/109 ... yes
75/110 ... yes
81/112 ... yes
82/113 ... yes
84/114 ... yes
86/115 ... yes
87/116 ... yes
88/118 ... yes
89/119 ... yes
87/120 ... yes
90/121 ... yes
91/122 ... yes
93/123 ... yes
94/124 ... yes
95/125 ... yes
96/126 ... yes
97/127 ... yes
98/128 ... yes
99/129 ... yes
100/130 ... yes
101/131 ... yes
102/132 ... yes
62/135 ... yes
92/136 ... yes
92/137 ... yes
92/138 ... yes
92/139 ... yes
92/140 ... yes
92/141 ... yes
92/142 ... yes
92/143 ... yes
92/144 ... yes
92/145 ... yes
92/146 ... yes
92/147 ... yes
92/148 ... yes
92/149 ... yes
92/150 ... yes
92/151 ... yes
92/152 ... yes
57/153 ... yes
92/154 ... yes
92/155 ... yes
92/156 ... yes
92/157 ... yes
92/158 ... yes
92/159 ... yes
92/160 ... yes
92/161 ... yes
92/162 ... yes
92/163 ... yes
92/164 ... yes
92/165 ... yes
92/166 ... yes
92/167 ... yes
92/168 ... yes
92/169 ... yes
92/170 ... yes
92/171 ... yes
92/172 ... yes
92/173 ... yes
92/174 ... yes
92/175 ... yes
92/176 ... yes
92/177 ... yes
92/178 ... yes
92/179 ... yes
92/180 ... yes
92/181 ... yes
92/182 ... yes
92/183 ... yes
92/184 ... yes
92/185 ... yes
92/186 ... yes
92/187 ... yes
92/188 ... yes
92/189 ... yes
92/190 ... yes
92/191 ... yes
92/192 ... yes
92/193 ... yes
92/194 ... yes
92/195 ... yes
92/196 ... yes
92/197 ... yes
92/198 ... yes
92/199 ... yes
92/200 ... yes
92/201 ... yes
92/202 ... yes
92/203 ... yes
92/204 ... yes
92/205 ... yes
92/206 ... yes
92/207 ... yes
92/208 ... yes
92/209 ... yes
92/210 ... yes
92/211 ... yes
92/212 ... yes
92/213 ... yes
92/214 ... yes
92/215 ... yes
92/216 ... yes
92/217 ... yes
92/218 ... yes
92/219 ... yes
92/220 ... yes
92/221 ... yes
92/222 ... yes
92/223 ... yes
92/224 ... yes
92/225 ... yes
92/226 ... yes
57/229 ... yes
69/230 ... yes
107/231 ... yes
106/232 ... yes
106/234 ... yes
57/235 ... yes
57/236 ... yes
109/238 ... yes
106/239 ... yes
109/240 ... yes
109/241 ... yes
109/242 ... yes
109/243 ... yes
53/244 ... yes
62/245 ... yes
111/246 ... yes
109/248 ... yes
109/249 ... yes
115/250 ... yes
115/251 ... yes
115/252 ... yes
62/254 ... yes
57/256 ... yes
50/257 ... yes
103/258 ... yes
118/260 ... yes
53/263 ... yes
62/264 ... yes
57/266 ... yes
57/269 ... yes
53/271 ... yes
62/272 ... yes
62/273 ... yes
57/274 ... yes
62/275 ... yes
57/276 ... yes
136/277 ... yes
53/278 ... yes
53/279 ... yes
57/280 ... yes
57/281 ... yes
57/282 ... yes
57/283 ... yes
57/284 ... yes
57/285 ... yes
57/288 ... yes
57/290 ... yes
57/291 ... yes
117/293 ... yes
62/294 ... yes
57/295 ... yes
57/296 ... yes
57/297 ... yes
57/298 ... yes
57/299 ... yes
57/300 ... yes
57/301 ... yes
57/302 ... yes
57/303 ... yes
57/304 ... yes
57/305 ... yes
57/306 ... yes
57/307 ... yes
57/308 ... yes
57/310 ... yes
57/311 ... yes
57/312 ... yes
57/313 ... yes
62/314 ... yes
57/315 ... yes
57/316 ... yes
57/317 ... yes
57/318 ... yes
57/319 ... yes
57/320 ... yes
138/321 ... yes
62/322 ... yes
109/323 ... yes
141/324 ... yes
141/325 ... yes
141/326 ... yes
141/327 ... yes
62/328 ... yes
62/329 ... yes
62/332 ... yes
53/333 ... yes
62/334 ... yes
53/335 ... yes
144/336 ... yes
138/337 ... yes
138/338 ... yes
62/339 ... yes
146/340 ... yes
146/341 ... yes
53/342 ... yes
62/343 ... yes
62/345 ... yes
144/346 ... yes
53/347 ... yes
53/348 ... yes
57/350 ... yes
144/351 ... yes
115/352 ... yes
57/353 ... yes
57/354 ... yes
57/355 ... yes
57/356 ... yes
57/357 ... yes
57/359 ... yes
57/360 ... yes
57/361 ... yes
57/362 ... yes
53/363 ... yes
53/369 ... yes
53/370 ... yes
53/371 ... yes
144/373 ... yes
144/374 ... yes
144/375 ... yes
144/376 ... yes
53/377 ... yes
144/380 ... yes
144/381 ... yes
144/382 ... yes
144/383 ... yes
144/384 ... yes
144/385 ... yes
144/386 ... yes
144/387 ... yes
144/388 ... yes
144/389 ... yes
144/390 ... yes
144/392 ... yes
144/393 ... yes
62/394 ... yes
153/395 ... yes
53/396 ... yes
106/397 ... yes
106/398 ... yes
53/399 ... yes
144/400 ... yes
50/401 ... yes
53/403 ... yes
57/404 ... yes
57/407 ... yes
57/408 ... yes
57/409 ... yes
57/410 ... yes
53/411 ... yes
53/412 ... yes
57/413 ... yes
53/414 ... yes
53/415 ... yes
159/417 ... yes
57/420 ... yes
53/421 ... yes
159/422 ... yes
53/431 ... yes
53/432 ... yes
53/435 ... yes
57/436 ... yes
162/437 ... yes
162/438 ... yes
53/440 ... yes
106/442 ... yes
170/443 ... yes
184/444 ... yes
185/445 ... yes
57/446 ... yes
57/447 ... yes
57/448 ... yes
53/449 ... yes
159/450 ... yes
174/451 ... yes
185/453 ... yes
174/454 ... yes
188/456 ... yes
188/457 ... yes
53/458 ... yes
57/459 ... yes
159/460 ... yes
159/461 ... yes
178/462 ... yes
53/463 ... yes
159/464 ... yes
174/466 ... yes
194/467 ... yes
159/468 ... yes
198/470 ... yes
197/471 ... yes
198/472 ... yes
178/474 ... yes
197/475 ... yes
199/476 ... yes
180/478 ... yes
180/479 ... yes
179/480 ... yes
197/482 ... yes
197/483 ... yes
197/484 ... yes
174/490 ... yes
174/491 ... yes
53/492 ... yes
53/493 ... yes
62/494 ... yes
153/496 ... yes
153/497 ... yes
178/499 ... yes
178/500 ... yes
178/501 ... yes
178/502 ... yes
179/503 ... yes
176/504 ... yes
180/507 ... yes
180/508 ... yes
159/509 ... yes
159/510 ... yes
178/512 ... yes
53/513 ... yes
53/514 ... yes
53/515 ... yes
176/516 ... yes
62/517 ... yes
53/518 ... yes
53/519 ... yes
115/520 ... yes
204/521 ... yes
204/522 ... yes
203/523 ... yes
203/524 ... yes
205/525 ... yes
42/526 ... yes
179/527 ... yes
194/528 ... yes
185/529 ... yes
185/530 ... yes
185/531 ... yes
203/532 ... yes
178/533 ... yes
53/534 ... yes
155/537 ... yes
209/538 ... yes
212/539 ... yes
213/540 ... yes
213/541 ... yes
203/542 ... yes
53/543 ... yes
53/544 ... yes
203/546 ... yes
216/547 ... yes
222/548 ... yes
222/549 ... yes
222/550 ... yes
57/551 ... yes
57/552 ... yes
57/553 ... yes
57/554 ... yes
57/555 ... yes
57/556 ... yes
202/557 ... yes
203/558 ... yes
203/559 ... yes
185/560 ... yes
226/561 ... yes
226/562 ... yes
226/563 ... yes
226/564 ... yes
226/565 ... yes
226/567 ... yes
203/568 ... yes
234/569 ... yes
53/570 ... yes
242/571 ... yes
242/572 ... yes
242/573 ... yes
242/574 ... yes
242/575 ... yes
242/576 ... yes
242/577 ... yes
242/578 ... yes
242/579 ... yes
242/580 ... yes
242/581 ... yes
243/582 ... yes
243/583 ... yes
243/585 ... yes
245/586 ... yes
245/587 ... yes
245/588 ... yes
245/589 ... yes
245/590 ... yes
242/591 ... yes
242/592 ... yes
246/593 ... yes
246/594 ... yes
246/595 ... yes
246/596 ... yes
246/597 ... yes
247/598 ... yes
247/599 ... yes
247/600 ... yes
247/601 ... yes
247/602 ... yes
248/603 ... yes
203/604 ... yes
245/605 ... yes
245/606 ... yes
245/607 ... yes
132/608 ... yes
138/609 ... yes
251/615 ... yes
178/616 ... yes
203/620 ... yes
248/625 ... yes
255/670 ... yes
255/671 ... yes
255/672 ... yes
258/673 ... yes
251/674 ... yes
251/675 ... yes
251/676 ... yes
251/677 ... yes
251/678 ... yes
251/679 ... yes
251/680 ... yes
251/682 ... yes
251/683 ... yes
251/684 ... yes
251/686 ... yes
251/687 ... yes
251/688 ... yes
251/689 ... yes
251/690 ... yes
251/691 ... yes
251/692 ... yes
251/693 ... yes
251/694 ... yes
251/695 ... yes
251/696 ... yes
251/697 ... yes
251/698 ... yes
251/699 ... yes
251/700 ... yes
251/701 ... yes
251/702 ... yes
251/703 ... yes
251/704 ... yes
251/705 ... yes
251/706 ... yes
251/707 ... yes
251/708 ... yes
251/709 ... yes
251/711 ... yes
203/712 ... yes
203/713 ... yes
259/714 ... yes
261/715 ... yes
132/716 ... yes
264/717 ... yes
197/718 ... yes
231/719 ... yes
142/720 ... yes
53/721 ... yes
251/722 ... yes
251/723 ... yes
251/732 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 83
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes