rake gitlab:backup:create broken on version 14.10

Summary

rake gitlab:backup:create failed on version 14.10

Steps to reproduce

# on arch linux
cd /usr/share/webapps/gitlab
sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:backup:create

Example Project

What is the current bug behavior?

The gitaly command failed after the dump of psql database

What is the expected correct behavior?

I expect the backup to works as expected.

Relevant logs and/or screenshots

$ sudo -u gitlab $(cat environment | xargs) bundle-2.7 exec rake gitlab:backup:create
2022-04-29 19:07:07 +0200 -- Dumping database ...
Dumping PostgreSQL database gitlab ... [DONE]
2022-04-29 19:07:09 +0200 -- Dumping database ... done
2022-04-29 19:07:09 +0200 -- Dumping repositories ...
flag provided but not defined: -layout
Usage of create:
  -locator string
    	determines how backup files are located. One of legacy, pointer. Note: The feature is not ready for production use. (default "legacy")
  -parallel int
    	maximum number of parallel backups (default 8)
  -parallel-storage -parallel
    	maximum number of parallel backups per storage. Note: actual parallelism when combined with -parallel depends on the order the repositories are received. (default 2)
  -path string
    	repository backup path
rake aborted!
Backup::Error: gitaly-backup exit status 2
/usr/share/webapps/gitlab/lib/backup/gitaly_backup.rb:58:in `finish!'
/usr/share/webapps/gitlab/lib/backup/repositories.rb:21:in `ensure in dump'
/usr/share/webapps/gitlab/lib/backup/repositories.rb:21:in `dump'
/usr/share/webapps/gitlab/lib/backup/manager.rb:82:in `run_create_task'
/usr/share/webapps/gitlab/lib/backup/manager.rb:46:in `block in create'
/usr/share/webapps/gitlab/lib/backup/manager.rb:45:in `each'
/usr/share/webapps/gitlab/lib/backup/manager.rb:45:in `create'
/usr/share/webapps/gitlab/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'

Caused by:
Errno::EPIPE: Broken pipe
/usr/share/webapps/gitlab/lib/backup/gitaly_backup.rb:86:in `write'
/usr/share/webapps/gitlab/lib/backup/gitaly_backup.rb:86:in `puts'
/usr/share/webapps/gitlab/lib/backup/gitaly_backup.rb:86:in `schedule_backup_job'
/usr/share/webapps/gitlab/lib/backup/gitaly_backup.rb:66:in `enqueue'
/usr/share/webapps/gitlab/lib/backup/repositories.rb:56:in `enqueue_project'
/usr/share/webapps/gitlab/lib/backup/repositories.rb:47:in `block in enqueue_consecutive_projects'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:71:in `block (2 levels) in find_each'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.7/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/webapps/gitlab/lib/backup/repositories.rb:46:in `enqueue_consecutive_projects'
/usr/share/webapps/gitlab/lib/backup/repositories.rb:41:in `enqueue_consecutive'
/usr/share/webapps/gitlab/lib/backup/repositories.rb:18:in `dump'
/usr/share/webapps/gitlab/lib/backup/manager.rb:82:in `run_create_task'
/usr/share/webapps/gitlab/lib/backup/manager.rb:46:in `block in create'
/usr/share/webapps/gitlab/lib/backup/manager.rb:45:in `each'
/usr/share/webapps/gitlab/lib/backup/manager.rb:45:in `create'
/usr/share/webapps/gitlab/lib/tasks/gitlab/backup.rake:12:in `block (3 levels) in <top (required)>'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/usr/share/webapps/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

System information
System:
Current User:	gitlab
Using RVM:	no
Ruby Version:	3.0.4p208
Gem Version:	/usr/lib/ruby/2.7.0/bundler/spec_set.rb:86:in `block in materialize': Could not find rake-13.0.6 in any of the sources (Bundler::GemNotFound)
	from /usr/lib/ruby/2.7.0/bundler/spec_set.rb:80:in `map!'
	from /usr/lib/ruby/2.7.0/bundler/spec_set.rb:80:in `materialize'
	from /usr/lib/ruby/2.7.0/bundler/definition.rb:170:in `specs'
	from /usr/lib/ruby/2.7.0/bundler/definition.rb:237:in `specs_for'
	from /usr/lib/ruby/2.7.0/bundler/definition.rb:226:in `requested_specs'
	from /usr/lib/ruby/2.7.0/bundler/runtime.rb:101:in `block in definition_method'
	from /usr/lib/ruby/2.7.0/bundler/runtime.rb:20:in `setup'
	from /usr/lib/ruby/2.7.0/bundler.rb:149:in `setup'
	from /usr/lib/ruby/2.7.0/bundler/setup.rb:20:in `block in '
	from /usr/lib/ruby/2.7.0/bundler/ui/shell.rb:136:in `with_level'
	from /usr/lib/ruby/2.7.0/bundler/ui/shell.rb:88:in `silence'
	from /usr/lib/ruby/2.7.0/bundler/setup.rb:20:in `'
	from :85:in `require'
	from :85:in `require'
Bundler Version:2.7.0
Rake Version:	13.0.6
Redis Version:	6.2.6
Sidekiq Version:6.4.0
Go Version:	go1.18.1 linux/amd64

GitLab information
Version:	14.10.0
Revision:	88da5554d96
Directory:	/usr/share/webapps/gitlab
DB Adapter:	PostgreSQL
DB Version:	14.2
URL:		https://myhost
HTTP Clone URL:	https://myhost/some-group/some-project.git
SSH Clone URL:	ssh://gitlab@myhost/some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	13.25.1
Repository storage paths:
- default: 	/var/lib/gitlab/repositories
GitLab Shell path:		/usr/share/webapps/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.25.1 ? ... OK (13.25.1)
Running /usr/share/webapps/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 0/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... no
Try fixing it:
Install the Service
For more information see:
doc/install/installation.md in section "Install the Service"
Please fix the error above and rerun the checks.
Systemd unit files or init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
2/1 ... yes
17/5 ... yes
3/7 ... yes
5/8 ... yes
4/9 ... yes
4/10 ... yes
6/11 ... yes
6/12 ... yes
7/13 ... yes
12/14 ... yes
8/15 ... yes
8/16 ... yes
8/17 ... yes
7/18 ... yes
10/20 ... yes
10/21 ... yes
11/22 ... yes
11/23 ... yes
12/24 ... yes
12/25 ... yes
13/26 ... yes
7/27 ... yes
7/28 ... yes
1/29 ... yes
16/31 ... yes
18/32 ... yes
12/33 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.6)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes