Object storage upload fails to Google Cloud Storage with Bucket Policy Only enabled
Summary
Uploads of artifacts (and all other types as well) fail to a Google Cloud Storage with the beta feature of Bucket Policy Only enabled.
Traditionally the Cloud IAM and the Storage ACL work together, any can grant access. This make it possible to define an ACL at upload time that allows public access to the uploaded object. No object in GitLab storages should ever be public and should only be reachable after proper authorization via a service account. The mentioned feature disables any ACL use and therefor stops this loophole of the IAM system from working.
Steps to reproduce
- Enable the "Bucket Policy Only" setting on a storage used to upload artifacts.
- Upload artifacts from a build.
What is the current bug behavior?
Artifact uploads fail:
WARNING: Uploading artifacts to coordinator... failed id=115 responseStatus=500 Internal Server Error status=500 Internal Server Error token=zHEADzfx
The log shows that the upload stuff tries to set an ACL, which is forbidden in this mode:
Google::Apis::ClientError (invalid: Cannot use ACL API to set object policy when object policies are disabled.):
What is the expected correct behavior?
Uploads succeed by not setting an unneeded ACL.
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 9.8 Current User: git Using RVM: no Ruby Version: 2.5.5p157 Gem Version: 2.5.2.1 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.6 Git Version: 2.20.1 Sidekiq Version:5.2.5 Go Version: go1.11.5 linux/amd64GitLab information Version: 11.9.0-rc8 Revision: 118c883e54 Directory: /srv/salsa-test.debian.net/gitlab DB Adapter: postgresql URL: https://salsa-test.debian.net HTTP Clone URL: https://salsa-test.debian.net/some-group/some-project.git SSH Clone URL: git@salsa-test.debian.net:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: bitbucket, github, gitlab, google_oauth2
GitLab Shell Version: 8.7.1 Repository storage paths:
- default: /srv/salsa-test.debian.net/repositories GitLab Shell path: /srv/salsa-test.debian.net/gitlab-shell Git: /usr/bin/git
Possible fixes
Currently carrierwave
forcefully sets stored objects to public or private, which triggers the ACL setup in fog-google
.