Backup upload to Google Cloud Storage fails because the `google-api-client` gem is missing

Summary

I am currently setting up a new GitLab instance on GCE and want to upload my backups to a Google Cloud Storage bucket.

I do not want to use S3 compatibility because the access credentials are always tied to a GCP project member's user account. Instead, I want to use Fog's native Google backend with a dedicated service account to access the bucket. Unfortunately, this is not possible because GitLab only bundles the fog-google gem, but not the google-api-client gem.

Steps to reproduce

  1. Install the GitLab Omnibus package
  2. Configure GitLab to upload backups to Google Cloud Storage:
gitlab_rails['backup_upload_connection'] = {
  'provider' => 'Google',
  'google_project' => 'my-project-id',
  'google_client_email' => 'gitlab-backup@my-project-id.iam.gserviceaccount.com',
  'google_json_key_location' => '/etc/gitlab/my-project.json'
}
gitlab_rails['backup_upload_remote_directory'] = 'my-gitlab-backup'
  1. Create a backup with gitlab-rake gitlab:backup:create

Expected behavior

GitLab should create a backup and upload it to the storage bucket.

Actual behavior

The upload fails because the google-api-client gem is missing.

Relevant logs and/or screenshots

# gitlab-rake gitlab:backup:create
Dumping database ... 
Dumping PostgreSQL database gitlabhq_production ... [DONE]
done
Dumping repositories ...
done
Dumping uploads ... 
done
Dumping builds ... 
done
Dumping artifacts ... 
done
Dumping pages ... 
done
Dumping lfs objects ... 
done
Dumping container registry images ... 
[DISABLED]
Creating backup archive: 1475743782_gitlab_backup.tar ... [fog][WARNING] Please install the google-api-client gem before using this provider
rake aborted!
Fog::Service::NotFound: google has no storage service
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:18:in `rescue in new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:8:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/storage.rb:27:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:157:in `connect_to_remote_directory'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:49:in `upload'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:36:in `block in pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `chdir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:21:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
LoadError: cannot load such file -- google/api_client
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `block in require'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:240:in `load_dependency'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-google-0.3.2/lib/fog/google/shared.rb:37:in `initialize_google_client'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-google-0.3.2/lib/fog/storage/google_json/real.rb:16:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/service.rb:115:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/service.rb:115:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-google-0.3.2/lib/fog/storage/google.rb:15:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/core/services_mixin.rb:16:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/fog-core-1.42.0/lib/fog/storage.rb:27:in `new'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:157:in `connect_to_remote_directory'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:49:in `upload'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:36:in `block in pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `chdir'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:19:in `pack'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:21:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)
done

Output of checks

Results of GitLab application Check

# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab Shell ...

GitLab Shell version >= 3.6.1 ? ... OK (3.6.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... no
  User id for git: 998. Groupd id for git: 998
  Try fixing it:
  sudo chown -R git:git /var/opt/gitlab/git-data/repositories
  For more information see:
  doc/install/installation.md in section "GitLab Shell"
  Please fix the error above and rerun the checks.
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... can't check, you have no projects
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP users with access to your GitLab server (only showing the first 100 results)
Server: ldapmain
	[Redacted]

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... no
  Try fixing it:
  sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \;
  sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \;
  For more information see:
  doc/install/installation.md in section "GitLab"
  Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ... can't check, you have no projects
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.7.4)
Active users: 1
Elasticsearch version >= 2.0? ... yes (2.1.2)
Elasticsearch has plugin delete-by-query installed? ... no

Checking GitLab ... Finished

Results of GitLab environment info

# gitlab-rake gitlab:env:info

System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.3.1p112
Gem Version:	2.6.6
Bundler Version:1.13.1
Rake Version:	10.5.0
Sidekiq Version:4.1.4

GitLab information
Version:	8.12.4-ee
Revision:	c810497
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	9.2.18
URL:		https://gitlab.mydomain.com
HTTP Clone URL:	https://gitlab.mydomain.com/some-group/some-project.git
SSH Clone URL:	git@gitlab-ssh.mydomain.com:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	yes
Using Omniauth:	no

GitLab Shell
Version:	3.6.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:		/opt/gitlab/embedded/bin/git

Possible fixes

Include the google-api-client gem 😄