Docs feedback: GitLab Container Registry administration Documentation Lacking (troubleshooting specifically)
In trying to follow the documentation for setting up the docker registry, on an omnibus installation, under a separate domain, it doesn't work. As a note, I do have this running behind an nginx reverse proxy, but it is listening on 80, 443, 4567 and 5000, and forwarding to the gitlab server ip. Also, I have LDAP enabled, but I have confirmed using the 'root' user to perform these steps also does not work.
The steps listed state:
All you have to do is configure the domain name under which the Container Registry will listen to. Read #container-registry-domain-configuration and pick one of the two options that fits your case.
You can either use the existing domain under a specific port, or a separate domain, the latter being what I'm trying, with these steps:
- Place your TLS certificate and key in /etc/gitlab/ssl/registry.gitlab.example.com.crt and /etc/gitlab/ssl/registry.gitlab.example.com.key and make sure they have correct permissions.
Which, i have done: (note, my cert TLD is '.services')
root@358844d397d1:/# ls -l /etc/gitlab/ssl/*
-rw------- 1 root root 3917 Mar 11 16:50 /etc/gitlab/ssl/docker.mydomain.services.crt
-rw------- 1 root root 3243 Mar 11 16:50 /etc/gitlab/ssl/docker.mydomain.services.key
- Once the TLS certificate is in place, edit /etc/gitlab/gitlab.rb with: registry_external_url 'https://registry.gitlab.example.com'
Which I have set as https://docker.mydomain.services
- Save the gitlab.rb and reconfigure with 'gitlab-ctl reconfigure'
and the added note:
Note: If you have a wildcard certificate, you need to specify the path to the certificate in addition to the URL, in this case /etc/gitlab/gitlab.rb will look like...
Which i have also done and saved before running reconfigure, pointing to the above listed files.
After all of this, running 'docker login docker.mydomain.services' or 'docker login https://docker.mydomain.services' results in:
Error response from daemon: Get https://docker.mydomain.services/v2/: denied: access forbidden
The troubleshooting section has two parts for:
- Using self-signed certificates with Container Registry
- AWS S3 with the GitLab registry error when pushing large images
I have spent a good amount of time trying to research this but still haven't been able to get it going. If I can figure out how to get this going, or if anyone else can list some steps for troubleshooting, I'd be happy to update the troubleshooting section and/or documentation and create a pull request!
So far, I can only suggest that I use netstat -tanpu | grep -i listen
to confirm that it's listening on the default external port:
tcp 0 0 0.0.0.0:4567 0.0.0.0:* LISTEN 25154/nginx tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN -
and monitoring the logs with tail -f /var/log/gitlab/gitlab-rails/production.log
Started GET "//jwt/auth?account=root&client_id=docker&offline_token=[FILTERED]&service=container_registry" for 127.0.0.1 at 2019-03-11 21:08:01 +0000
Processing by JwtController#auth as HTML
Parameters: {"account"=>"root", "client_id"=>"docker", "offline_token"=>"[FILTERED]", "service"=>"container_registry"}
Completed 403 Forbidden in 4ms (Views: 0.2ms | ActiveRecord: 0.0ms)
and tail -f /var/log/gitlab/registry/current
2019-03-11_21:15:20.10351 time="2019-03-11T21:15:20.102669193Z" level=debug msg="authorizing request" environment=production go.version=go1.10.3 http.request.host=docker.mydomain.services http.request.id=f9fed741-ee72-4c60-972c-41f123392d67 http.request.method=GET http.request.remoteaddr=10.255.0.2 http.request.uri="/v2/" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.2 \\(windows\\))" instance.id=311e3751-8a13-4805-96b2-422d4c4c95a7 service=registry version=v2.6.0-rc.1-158-gb22c6b7
2019-03-11_21:15:20.10354 time="2019-03-11T21:15:20.102737692Z" level=warning msg="error authorizing context: authorization token required" environment=production go.version=go1.10.3 http.request.host=docker.mydomain.services http.request.id=f9fed741-ee72-4c60-972c-41f123392d67 http.request.method=GET http.request.remoteaddr=10.255.0.2 http.request.uri="/v2/" http.request.useragent="docker/18.09.2 go/go1.10.6 git-commit/6247962 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.2 \\(windows\\))" instance.id=311e3751-8a13-4805-96b2-422d4c4c95a7 service=registry version=v2.6.0-rc.1-158-gb22c6b7
2019-03-11_21:15:20.10357 127.0.0.1 - - [11/Mar/2019:21:15:20 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/18.09.2 go/go1.10.6 git-commit/6247962 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.2 \\(windows\\))"