Registry with s3 storage behind proxy, cannot access s3 bucket.

Summary

I have a Gitlab host behind a corporate proxy, and try to enable the registry with s3 storage driver. When running a job needing to push a docker image, the push job is stuck at preparing the image. I can see that gitlab tries to access a public s3 url to connect to the bucket, instead of using the corporate http proxy.

Am I missing some parameters?

Steps to reproduce

• Run Gitlab behind corporate proxy
• Configure gitlab registry with s3 storage
• Try to push a docker image to the registry

What is the current bug behavior?

• Authentication to the registry succeeds
• Gitlab host tries to upload straight to the public s3 url instead of using proxy, which causes a push timeout.

What is the expected correct behavior?

Using the proxy to push image from Gitlab to s3 bucket.

Relevant logs

tcpdump extract:

12:43:41.459962 IP ip-X-X-X-X.eu-west-1.compute.internal.51980 > s3-eu-west-1-r-w.amazonaws.com.https: Flags [S], seq 1389189044, win 26883, options [mss 8961,sackOK,TS val 726071869 ecr 0,nop,wscale 7], length 0

docker log:

The push refers to repository [redacted]
918e6ad3de17: Preparing
ae4ead0cbdc6: Preparing
d51fcd4bb916: Preparing
40a06d506acd: Preparing
32bc3aa847b6: Preparing
796be3c21b05: Preparing
9698defc253a: Preparing
2903a73e7a0d: Preparing
409170aec809: Preparing
2e669e0134f5: Preparing
8bacec4e3446: Preparing
26b1991f37bd: Preparing
55e6b89812f3: Preparing
796be3c21b05: Waiting
9698defc253a: Waiting
2903a73e7a0d: Waiting
409170aec809: Waiting
2e669e0134f5: Waiting
26b1991f37bd: Waiting
8bacec4e3446: Waiting
55e6b89812f3: Waiting
32bc3aa847b6: Retrying in 5 seconds
40a06d506acd: Retrying in 5 seconds
ae4ead0cbdc6: Retrying in 5 seconds
918e6ad3de17: Retrying in 5 seconds
d51fcd4bb916: Retrying in 5 seconds

Details of package version

gitlab-ee 12.3.1-ee.0

Environment details

• Operating System: Ubuntu 18.04
• Installation Target:
  • EC2 instance
• Installation Type:
  • New Installation
• Is there any other software running on the machine: No
• Is this a single or multiple node installation? Single node
• Resources
  • m5.large instance

Configuration details

gitlab_rails['env'] = {
        "http_proxy" => "http://proxy.redacted:8080",
        "https_proxy" => "http://proxy.redacted:8080",
        "no_proxy" => "localhost,127.0.0.1,.redacted"}
gitaly['env'] = {
    "http_proxy" => "http://proxy.redacted:8080",
    "https_proxy" => "http://proxy.redacted:8080",
    "no_proxy" => "unix,localhost,127.0.0.1,.redacted"

gitlab_workhorse['env'] = {
    "http_proxy" => "http://proxy.redacted:8080",
    "https_proxy" => "http://proxy.redacted:8080",
    "no_proxy" => "unix,localhost,127.0.0.1,.redacted"
}
registry['env'] = {
    "http_proxy" => "http://proxy.redacted",
    "https_proxy" => "http://proxy.redacted",
    "no_proxy" => "unix,localhost,127.0.0.1,.redacted"
}

 registry['storage'] = {
   's3' => {
     'accesskey' => 'AKIREDACTED',
     'secretkey' => 'redacted',
     'region' => 'eu-west-1',
     'bucket' => 'redacted'
   }
 }

Thanks for your help!