SSL errors with the Docker container registry

When setting up the Docker container registry with the GDK, there doesn't seem to be a proper way to configure the SSL certificates. I ran into the following error when visiting the container registry page on a project(localhost:3000/namespace/some-project/container_registry).

Faraday::SSLError in Projects::Registry::RepositoriesController#index
SSL_connect returned=1 errno=0 state=error: certificate verify failed

See the "Further workaround" section below for more info on how we worked around it.

Some relevant documentation

Using a secure HTTPS/SSL registry

Generate certs (via https://serverfault.com/a/845788/182342)

openssl req \
  -newkey rsa:2048 \
  -x509 \
  -nodes \
  -keyout /Users/eric/my-ce-docker-registry/my-certs/localhost.key \
  -new \
  -out /Users/eric/my-ce-docker-registry/my-certs/localhost.crt \
  -subj /CN=localhost \
  -reqexts SAN \
  -extensions SAN \
  -config <(cat /System/Library/OpenSSL/openssl.cnf \
      <(printf '[SAN]\nsubjectAltName=DNS:localhost')) \
  -sha256 \
  -days 3650

Using Docker for Mac 17.06.2-ce-mac27. After this command you should be able to visit https://localhost:5063/v2/_catalog and get {"repositories":[]}

docker run -d \
  -p 5063:5000 \
  --restart=always \
  --name registry-ce \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/localhost.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/localhost.key \
  -v /Users/eric/my-ce-docker-registry:/var/lib/registry \
  -v /Users/eric/my-ce-docker-registry/my-certs/:/certs \
  registry:2

$gdk/gitlab/config/gitlab.yml

registry:
  enabled: true
  host: localhost
  port: 5063
  api_url: https://localhost:5063/ # internal address to the registry, will be used by GitLab to directly communicate with API
  key: /Users/eric/my-ce-docker-registry/my-certs/localhost.key
  path: Users/eric/my-ce-docker-registry/
  # not sure what the issuer would be???
  # issuer: gitlab-issuer

Visiting the container registry page on a project(localhost:3000/namespace/some-project/container_registry) will show the Faraday::SSLError now.

Further workaround

@stanhu mentioned this workaround which mimics how we setup things in omnibus.

config/initializers/1_settings.rb

# Works
ENV['SSL_CERT_FILE'] = "/Users/eric/my-ce-docker-registry/my-certs/localhost.crt"

# or ...

# Does not work
ENV['SSL_CERT_DIR'] = "/Users/eric/my-ce-docker-registry/my-certs/"

@stanhu also linked some more documentation about Faraday and SSL certs. I was unable to use the OPENSSLDIR(from openssl version -a) because /System/Library/OpenSSL/certs is protected with macOS "System Integrity Protection" which means I can't copy anything into it even with sudo (perhaps there is some special command to get things in there 😕). Even so, it seems that you need to configure Faraday with that directory so we could just to point it to an easier directory to work with.

I also tried the following because it seemed related but didn't seem to make a difference, https://docs.gitlab.com/ce/install/installation.html#using-https

$gdk/gitlab-shell/config.yml

http_settings:
  // ...
  ca_file: /Users/eric/my-ce-docker-registry/my-certs/localhost.crt
  // or
  ca_path: /Users/eric/my-ce-docker-registry/my-certs/

Is there a more concrete way to get things working?

Potential Solutions

We could add $gdk/gitlab/config/gitlab.yml -> registry -> cert that would setup the SSL_CERT_FILE environment variable manually.

It's not quite clear to me how we handle multiple certs (say one for the main GitLab domain and another for the registry) if you can only define a single cert via SSL_CERT_FILE(SSL_CERT_DIR didn't work for me). Omnibus looks to have a mechanism to handle multiple via /etc/gitlab/trusted-certs/.

Using an Insecure registry

Update: See https://gitlab.com/snippets/73109#setup-the-container-registry for a full correct way to set this up.

I tried using an insecure(http) registry but GitLab throws an error about the key missing from the config.

TypeError in Projects::Registry::RepositoriesController#index
no implicit conversion of nil into String

    def key_data
->        @key_data ||= File.read(key_file)
    end

Setup

docker run -d \
  -p 5063:5000 \
  --restart=always \
  --name registry-ce \
  registry:2

Configure the Docker Daemon to allow pushing to the insecure registry. With Docker for Mac, from the menubar icon -> Preference -> Daemon -> Insecure Registries -> Add http://localhost:5063 -> Click the "Apply & Restart" button

$gdk/gitlab/config/gitlab.yml

registry:
  enabled: true
  host: localhost
  port: 5063
  api_url: http://localhost:5063/ # internal address to the registry, will be used by GitLab to directly communicate with API

Reference, https://gitlab.slack.com/archives/C02PF508L/p1506031313000134

cc @stanhu @ayufan

Edited by Eric Eastwood