Azure Blob Rake backup fails
Summary
Using sudo /opt/gitlab/bin/gitlab-backup create
I am trying to create a backup to upload to an Azure container, but these fails when it reaches the upload stage with the error:
Uploading backup archive to remote storage gitlab ... rake aborted!
ArgumentError: Either optional parameter 'content_length' should be set or 'content' should implement 'size' method to get payload's size.
Steps to reproduce
- Install gitlab-ce (13.4.2-ce.0) on Ubuntu 16.04
- Configure /etc/gitlab/gitlab.rb with the Azure information (security keys redacted):
gitlab_rails['backup_upload_connection'] = {
'provider' => 'AzureRM',
'azure_storage_account_name' => '[Account Name]',
'azure_storage_access_key' => '[Access Key]',
'azure_storage_domain' => 'blob.core.usgovcloudapi.net', # US Government Domain
}
gitlab_rails['backup_upload_remote_directory'] = '[Directory Name]'
- Run
sudo gitlab-ctl reconfigure
- Run
sudo /opt/gitlab/bin/gitlab-backup create --trace
Example Project
Not really possible.
What is the current bug behavior?
The backup job fails with
...
Creating backup archive: 1601586181_2020_10_01_13.4.2_gitlab_backup.tar ... done
Uploading backup archive to remote storage gitlab ... rake aborted!
ArgumentError: Either optional parameter 'content_length' should be set or 'content' should implement 'size' method to get payload's size.
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/azure-storage-blob-2.0.0/lib/azure/storage/blob/block.rb:104:in `create_block_blob'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-fog-azure-rm-0.9.0/lib/fog/azurerm/requests/storage/create_block_blob.rb:31:in `create_block_blob'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-fog-azure-rm-0.9.0/lib/fog/azurerm/requests/storage/multipart_save_block_blob.rb:62:in `multipart_save_block_blob'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-fog-azure-rm-0.9.0/lib/fog/azurerm/models/storage/file.rb:254:in `save_blob'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-fog-azure-rm-0.9.0/lib/fog/azurerm/models/storage/file.rb:95:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:50:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:52:in `upload'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:26:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create
What is the expected correct behavior?
The backup job completes successfully and uploads the backup to the Azure Blob.
Relevant logs and/or screenshots
$ sudo /opt/gitlab/bin/gitlab-backup create --trace
** Invoke gitlab:backup:create (first_time)
** Invoke gitlab_environment (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute gitlab_environment
** Execute gitlab:backup:create
** Invoke gitlab:backup:db:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:db:create
2020-10-01 21:01:33 +0000 -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [DONE]
2020-10-01 21:01:36 +0000 -- done
** Invoke gitlab:backup:repo:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:repo:create
2020-10-01 21:01:36 +0000 -- Dumping repositories ...
* [REDACTED] (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ...
* [REDACTED] (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... [DONE]
* [REDACTED] (@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35) ... [SKIPPED] Wiki
* [REDACTED] (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ...
* [REDACTED] (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... [DONE]
* [REDACTED] (@hashed/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a) ... [SKIPPED] Wiki
* [REDACTED] (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ...
* [REDACTED] (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... [DONE]
* [REDACTED] (@hashed/e7/f6/e7f6c011776e8db7cd330b54174fd76f7d0216b612387a5ffcfb81e6f0919683) ... [SKIPPED] Wiki
* [REDACTED] (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ...
* [REDACTED] (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... [DONE]
* [REDACTED] (@hashed/79/02/7902699be42c8a8e46fbbb4501726517e86b22c56a189f7625a6da49081b2451) ... [DONE] Wiki
* [REDACTED] (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ...
* [REDACTED] (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... [DONE]
* [REDACTED] (@hashed/2c/62/2c624232cdd221771294dfbb310aca000a0df6ac8b66b696d90ef06fdefb64a3) ... [SKIPPED] Wiki
* [REDACTED] (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ...
* [REDACTED] (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... [DONE]
* [REDACTED] (@hashed/19/58/19581e27de7ced00ff1ce50b2047e7a567c76b1cbaebabe5ef03f7c3017bb5b7) ... [SKIPPED] Wiki
* [REDACTED] (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ...
* [REDACTED] (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... [DONE]
* [REDACTED] (@hashed/4a/44/4a44dc15364204a80fe80e9039455cc1608281820fe2b24f1e5233ade6af1dd5) ... [SKIPPED] Wiki
* [REDACTED] (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ...
* [REDACTED] (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... [DONE]
* [REDACTED] (@hashed/4f/c8/4fc82b26aecb47d2868c4efbe3581732a3e7cbcc6c2efb32062c08170a05eeb8) ... [SKIPPED] Wiki
* [REDACTED] (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ...
* [REDACTED] (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [DONE]
* [REDACTED] (@hashed/45/23/4523540f1504cd17100c4835e85b7eefd49911580f8efff0599a8f283be6b9e3) ... [SKIPPED] Wiki
2020-10-01 21:02:50 +0000 -- done
** Invoke gitlab:backup:uploads:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:uploads:create
2020-10-01 21:02:50 +0000 -- Dumping uploads ...
2020-10-01 21:02:50 +0000 -- done
** Invoke gitlab:backup:builds:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:builds:create
2020-10-01 21:02:50 +0000 -- Dumping builds ...
2020-10-01 21:02:50 +0000 -- done
** Invoke gitlab:backup:artifacts:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:artifacts:create
2020-10-01 21:02:50 +0000 -- Dumping artifacts ...
2020-10-01 21:02:50 +0000 -- done
** Invoke gitlab:backup:pages:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:pages:create
2020-10-01 21:02:50 +0000 -- Dumping pages ...
2020-10-01 21:02:50 +0000 -- done
** Invoke gitlab:backup:lfs:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:lfs:create
2020-10-01 21:02:50 +0000 -- Dumping lfs objects ...
2020-10-01 21:03:01 +0000 -- done
** Invoke gitlab:backup:registry:create (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:registry:create
2020-10-01 21:03:01 +0000 -- Dumping container registry images ...
2020-10-01 21:03:01 +0000 -- done
Creating backup archive: 1601586181_2020_10_01_13.4.2_gitlab_backup.tar ... done
Uploading backup archive to remote storage gitlab ... rake aborted!
ArgumentError: Either optional parameter 'content_length' should be set or 'content' should implement 'size' method to get payload's size.
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/azure-storage-blob-2.0.0/lib/azure/storage/blob/block.rb:104:in `create_block_blob'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-fog-azure-rm-0.9.0/lib/fog/azurerm/requests/storage/create_block_blob.rb:31:in `create_block_blob'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-fog-azure-rm-0.9.0/lib/fog/azurerm/requests/storage/multipart_save_block_blob.rb:62:in `multipart_save_block_blob'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-fog-azure-rm-0.9.0/lib/fog/azurerm/models/storage/file.rb:254:in `save_blob'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-fog-azure-rm-0.9.0/lib/fog/azurerm/models/storage/file.rb:95:in `save'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/fog-core-2.1.0/lib/fog/core/collection.rb:50:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:52:in `upload'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:26:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 16.04
Current User: git
Using RVM: no
Ruby Version: 2.6.6p146
Gem Version: 2.7.10
Bundler Version:1.17.3
Rake Version: 12.3.3
Redis Version: 5.0.9
Git Version: 2.28.0
Sidekiq Version:5.2.9
Go Version: unknown
GitLab information
Version: 13.4.2
Revision: b08b36dccc3
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 11.9
URL: https://<url>
HTTP Clone URL: https://<url>/some-group/some-project.git
SSH Clone URL: git@<url>:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.7.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
Possible fixes
Not sure. I tried adding
gitlab_rails['backup_multipart_chunk_size'] = 104857600
which didn't seem to matter