gitaly-backup exit status 1 after update to 17.4.1
Summary
Trying to do back up using
docker exec -it gitlab /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1 SKIP=lfs
results in rake aborted.
Steps to reproduce
$ docker exec gitlab /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1 SKIP=lfs
rake aborted!
Backup::Error: gitaly-backup exit status 1
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/gitaly_backup.rb:49:in `finish!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/targets/repositories.rb:33:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/tasks/task.rb:25:in `backup!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:57:in `run_create_task'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:145:in `block in run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:145:in `each_value'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:145:in `run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:32:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:12:in `block in create_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:75:in `lock_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:10:in `create_backup'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:117:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:25:in `load'
/opt/gitlab/embedded/bin/bundle:25:in `<main>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)What is the current bug behavior?
backup fails
What is the expected correct behavior?
Backup should complete.
Relevant logs
Relevant logs
Nothing shows up abort rails abort.
==> /var/log/gitlab/gitlab-rails/backup_json.log <==
{"severity":"INFO","time":"2024-09-26T18:58:47.616Z","correlation_id":null,"message":"Dumping PostgreSQL database gitlabhq_production ... "}
{"severity":"INFO","time":"2024-09-26T18:59:03.235Z","correlation_id":null,"message":"[DONE]"}
{"severity":"INFO","time":"2024-09-26T18:59:03.235Z","correlation_id":null,"message":"Dumping database ... done"}
{"severity":"INFO","time":"2024-09-26T18:59:03.236Z","correlation_id":null,"message":"Dumping repositories ... "}
{"severity":"INFO","time":"2024-09-26T19:01:22.690Z","correlation_id":null,"message":"Deleting tar staging files ... "}
{"severity":"INFO","time":"2024-09-26T19:01:22.692Z","correlation_id":null,"message":"Cleaning up /var/opt/gitlab/backups/db"}
{"severity":"INFO","time":"2024-09-26T19:01:22.718Z","correlation_id":null,"message":"Cleaning up /var/opt/gitlab/backups/repositories"}
{"severity":"INFO","time":"2024-09-26T19:01:27.085Z","correlation_id":null,"message":"Deleting tar staging files ... done"}
{"severity":"INFO","time":"2024-09-26T19:01:27.085Z","correlation_id":null,"message":"Deleting backups/tmp ... "}
{"severity":"INFO","time":"2024-09-26T19:01:27.086Z","correlation_id":null,"message":"Deleting backups/tmp ... done"}
==> /var/log/gitlab/gitlab-rails/audit_json.log <==
{"severity":"INFO","time":"2024-09-26T12:09:58.758Z","correlation_id":"01J8Q3SVA5RZHZHYZBYCZB1A7G","meta.caller_id":"SessionsController#create","meta.feature_c
ategory":"system_access","meta.remote_ip":"172.17.0.1","meta.user":"jearisman","meta.user_id":184,"meta.client_id":"user/184","author_id":184,"author_name":"Je
sse Earisman","entity_id":184,"entity_type":"User","created_at":"2024-09-26T12:09:58.757+00:00","with":"standard","target_id":184,"target_type":"User","target_
details":"Jesse Earisman"}
{"severity":"INFO","time":"2024-09-26T13:11:17.786Z","correlation_id":"01J8Q7A4NJGKW0C7SQKBW0NM7R","meta.caller_id":"SessionsController#create","meta.feature_c
ategory":"system_access","meta.remote_ip":"172.17.0.1","meta.user":"conner.medeiros","meta.user_id":215,"meta.client_id":"user/215","author_id":215,"author_nam
e":"Conner Medeiros","entity_id":215,"entity_type":"User","created_at":"2024-09-26T13:11:17.786+00:00","with":"standard","target_id":215,"target_type":"User","
target_details":"Conner Medeiros"}
{"severity":"INFO","time":"2024-09-26T13:28:36.029Z","correlation_id":"01J8Q89TDDMW2ZZWWJMVDXNJHP","meta.caller_id":"SessionsController#create","meta.feature_c
ategory":"system_access","meta.remote_ip":"172.17.0.1","meta.user":"hshim","meta.user_id":64,"meta.client_id":"user/64","author_id":64,"author_name":"Hyong Shi
m","entity_id":64,"entity_type":"User","created_at":"2024-09-26T13:28:36.028+00:00","with":"standard","target_id":64,"target_type":"User","target_details":"Hyo
ng Shim"}
Details of package version
Provide the package version installation details
external_url 'https://nash-gitlab.****labs.com' gitlab_rails['gitlab_email_enabled'] = true gitlab_rails['gitlab_email_from'] = 'admin@****labs.com' gitlab_rails['gitlab_email_display_name'] = 'NASH Gitlab' gitlab_rails['gitlab_email_reply_to'] = 'noreply-gitlab@****labs.com' gitlab_rails['mathjax_load'] = 'always' gitlab_rails['gitlab_shell_ssh_port'] = 22 gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = "thumper.****labs.com" gitlab_rails['smtp_port'] = 25 gitlab_rails['smtp_domain'] = "labs.com" gitlab_rails['smtp_enable_starttls_auto'] = false gitlab_rails['smtp_tls'] = false gitlab_rails['smtp_openssl_verify_mode'] = 'none' postgresql['shared_buffers'] = "4GB" # recommend value is 1/4 of total RAM, up to 14GB. postgresql['max_connections'] = 901 nginx['redirect_http_to_https'] = true nginx['custom_gitlab_server_config'] = "location /-nightly/ {\n proxy_pass http://127.0.0.1:9594/;\n proxy_cache_valid 200 1m;\n}\n"
~ ~