The backup upload on Google Cloud Storage fails if the backup is very large
Hi all, I have configured my backup in this way
gitlab_rails['backup_upload_connection'] = {
'provider' => 'Google',
'google_project' => 'my-gcp-project',
'google_client_email' => 'my-account@my-gcp-project.iam.gserviceaccount.com',
'google_json_key_location' => '/mylocation/my-gcp-project.json'
}
gitlab_rails['backup_upload_remote_directory'] = 'mybucket'
gitlab_rails['backup_multipart_chunk_size'] = 104857600
(following what I found here )
If the backup file is less than 1GB I have no problem and it is uploaded on GCP.
When the backup is bigger than 1GB the upload fails (I tried with 17GB... but my full backup is 100GB)
Here below the log when backup fails:
$ gitlab-rake gitlab:backup:create STRATEGY=copy SKIP=db,uploads,repositories,builds,lfs,registry
Dumping database ...
[SKIPPED]
Dumping repositories ...
[SKIPPED]
Dumping uploads ...
[SKIPPED]
Dumping builds ...
[SKIPPED]
Dumping artifacts ...
done
Dumping pages ...
done
Dumping lfs objects ...
[SKIPPED]
Dumping container registry images ...
[DISABLED]
Creating backup archive: 1557261721_2019_05_07_11.7.5-ee_gitlab_backup.tar ... done
Uploading backup archive to remote storage mybucket ... rake aborted!
Google::Apis::TransmissionError: execution expired
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:50:in `upload'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:35:in `block in pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `chdir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:20:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
HTTPClient::SendTimeoutError: execution expired
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:50:in `upload'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:35:in `block in pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `chdir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:20:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)
and here a working backup with a small file:
$gitlab-rake gitlab:backup:create STRATEGY=copy SKIP=db,uploads,repositories,builds,artifacts,lfs,registry
Dumping database ...
[SKIPPED]
Dumping repositories ...
[SKIPPED]
Dumping uploads ...
[SKIPPED]
Dumping builds ...
[SKIPPED]
Dumping artifacts ...
[SKIPPED]
Dumping pages ...
done
Dumping lfs objects ...
[SKIPPED]
Dumping container registry images ...
[DISABLED]
Creating backup archive: 1557262811_2019_05_07_11.7.5-ee_gitlab_backup.tar ... done
Uploading backup archive to remote storage mybucket ...
done
Deleting tmp directories ... done
done
Deleting old backups ... skipping
~bug ~"backup-restore"