Cannot fetch the job log when using Job artifacts Object Store

Summary

We are currently testing the artifacts object store, on our private gitlab-ee deployement Our CI pipelines works, files are uploaded correctly While the job is in progress, logs can be read on the web interface However, we cannot read the job's log once the job is complete : a red box "An error occurred while fetching the job log." appears

On the object store's access log, we have those:

2019-01-28 12:27:13.084 7fb5282e5700  1 civetweb: 0x55a2561b4000: 10.11.45.61 - - [28/Jan/2019:12:27:13 +0100] "PUT /artifacts/90/b0/90b0ce469fbd8e30a2862bb24d562dc641c534a9b43c7c33c25cfaefe25e5e47/2019_01_28/168804/186957/job.log HTTP/1.1" 200 209 - fog-core/1.45.0

And when we check out the log, we get those:

2019-01-28 12:51:20.906 7fb5282e5700  1 civetweb: 0x55a2561b4000: 10.11.45.61 - - [28/Jan/2019:12:51:20 +0100] "GET /artifacts/21/90/21900f41ecb7b8e6cfd9250f096aad2fe7f6d8fbec9436b2d28e48c304ff8255/2019_01_28/168799/186950/job.log?X-Amz-Expires=600&X-Amz-Date=20190128T115120Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=xxx/20190128/eu-central-1/s3/aws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=xxx HTTP/1.1" 403 407 - Ruby

Steps to reproduce

Enable artifacts_enabled and artifacts_object_store_enabled, using the following config:

gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_direct_upload'] = false
gitlab_rails['artifacts_object_store_background_upload'] = true
gitlab_rails['artifacts_object_store_proxy_download'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts"
gitlab_rails['artifacts_object_store_connection'] = {
       'provider' => 'AWS',
       'host' => 'our_fqdn',
       'endpoint' => 'https://our_fqdn',
       'region' => 'eu-central-1',
       'path_style' => true,
       'aws_access_key_id' => 'access_key',
       'aws_secret_access_key' => 'secret_key'
}

What is the current bug behavior?

We cannot get a jobs log once it is completed

What is the expected correct behavior?

We should be able to see those logs

Results of GitLab environment info

Expand for output related to GitLab environment info 
sudo gitlab-rake gitlab:env:info

System information
System:		Debian 9.7
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.5.3p105
Gem Version:	2.7.6
Bundler Version:1.16.6
Rake Version:	12.3.2
Redis Version:	3.2.12
Git Version:	2.18.1
Sidekiq Version:5.2.3
Go Version:	unknown


GitLab information
Version:	11.7.0-ee
Revision:	c02f0d4
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
DB Version:	9.6.11
URL:		https://gitlab.oxygem.tv
HTTP Clone URL:	https://gitlab.oxygem.tv/some-group/some-project.git
SSH Clone URL:	git@gitlab.oxygem.tv:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	8.4.4
Repository storage paths:
default: 	/var/opt/gitlab/git-data/repositories
Hooks:		/opt/gitlab/embedded/service/gitlab-shell/hooks
Git:		/opt/gitlab/embedded/bin/git
Edited by Elliot Rushton