After the 9.4.0-ee update integration with a remote docker registry is no longer working;
The "Registry" part of the repository is showing "No tags in Container Registry for this container image."
Meanwhile, all the jobs are failing with:
Running with gitlab-ci-multi-runner 9.4.0 (ef0b1a6)
on gitlab-runner-0001 (9b3f1890)
Using Docker executor with image docker:latest ...
Using docker image sha256:5b8d5c26c13ca247dbde59d4a5c46d858aa2cc79da12e3a34aa8bc46a1eeb75b for predefined container...
Pulling docker image docker:latest ...
Using docker image docker:latest ID=sha256:192e3edb771f334fbc2e1941b22a0b3ecd8545a86744ce8fc2fe98a3d5774273 for build container...
Running on runner-9b3f1890-project-65-concurrent-0 via gitlab-runner-0001...
Cloning repository...
Cloning into '/builds/applaudience/go2cinema-com-web-app'...
Checking out 7e718c7a as master...
Skipping Git submodules setup
$ docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.anuary.com
Error response from daemon: login attempt to https://registry.anuary.com/v2/ failed with status: 401 Unauthorized
ERROR: Job failed: exit code 1
For additional debugging context:
$ export PASSWORD="REDACTED"
$ export REGISTRY="https://registry.anuary.com/"
$ export TOKEN=$(curl --silent --user ${USERNAME}:${PASSWORD} 'https://git.anuary.com/jwt/auth?service=container_registry&scope=registry:*:*' | jq -r '.token')
$ curl -s -H "Authorization: Bearer ${TOKEN}" ${REGISTRY}v2/_catalog
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}
Further investigation shows "invalid_token":
curiosity~ % curl -Lv -H "Authorization: Bearer ${TOKEN}" https://registry.anuary.com/v2/
* Trying 51.15.60.210...
* TCP_NODELAY set
* Connected to registry.anuary.com (51.15.60.210) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: registry.anuary.com
* Server certificate: Let's Encrypt Authority X3
* Server certificate: DST Root CA X3
> GET /v2/ HTTP/1.1
> Host: registry.anuary.com
> User-Agent: curl/7.51.0
> Accept: */*
> Authorization: Bearer REDACTED
>
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< Www-Authenticate: Bearer realm="https://git.anuary.com/jwt/auth",service="container_registry",error="invalid_token"
< X-Content-Type-Options: nosniff
< Date: Sun, 23 Jul 2017 14:03:08 GMT
< Content-Length: 87
<
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}
* Curl_http_done: called premature == 0
* Connection #0 to host registry.anuary.com left intact
Just as a sanity check, I've check /var/log/yum.log. The only thing to have changed in the last 24 hours is the gitlab-ee and the multi-runner:
Jul 21 07:42:41 Updated: gitlab-ee.x86_64 9.3.9-ee.0.el7
Jul 23 06:44:21 Updated: gitlab-ci-multi-runner.x86_64 9.4.0-1
Jul 23 06:45:40 Updated: gitlab-ee.x86_64 9.4.0-ee.0.el7
Looking into the docker logs brought up the error that "token signed by untrusted key with ID":
35.187.184.108 - - [23/Jul/2017:14:13:07 +0000] "GET /v1/repositories/applaudience/medium-community-manager/images HTTP/1.1" 404 19 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2017-07-23T14:13:44Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.7.6 http.request.host=registry.anuary.com http.request.id=6b9ad336-29a1-4e5f-9b9d-32c699274cc0 http.request.method=GET http.request.remoteaddr="35.187.184.108:33652" http.request.uri="/v2/" http.request.useragent="docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" instance.id=ac1e033e-e8e6-4ff1-be8b-e6dfbaa3273d version=v2.6.2
35.187.184.108 - - [23/Jul/2017:14:13:44 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
time="2017-07-23T14:13:44Z" level=info msg="token signed by untrusted key with ID: \"7K7M:VDNO:YAOL:X4ST:E2XK:HYLF:ROY4:7L7L:SCLH:46TJ:W3LZ:FJCH\""
time="2017-07-23T14:13:44Z" level=warning msg="error authorizing context: invalid token" go.version=go1.7.6 http.request.host=registry.anuary.com http.request.id=6c9a0157-4894-43c4-9705-b68e3e64ca7f http.request.method=GET http.request.remoteaddr="35.187.184.108:33658" http.request.uri="/v2/applaudience/forward-proxy/manifests/1933.b249d25a3d06f76768b2bbc66e7ce3159aee4d3e" http.request.useragent="docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" instance.id=ac1e033e-e8e6-4ff1-be8b-e6dfbaa3273d vars.name="applaudience/forward-proxy" vars.reference=1933.b249d25a3d06f76768b2bbc66e7ce3159aee4d3e version=v2.6.2
35.187.184.108 - - [23/Jul/2017:14:13:44 +0000] "GET /v2/applaudience/forward-proxy/manifests/1933.b249d25a3d06f76768b2bbc66e7ce3159aee4d3e HTTP/1.1" 401 169 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(Go-http-client/1.
Edited by Gajus Kuizinas