[Registry] Can't pull images but could push.
Summary
Registry could push images to it but can't pull images from it.
Steps to reproduce
-
Install Gitlab with Charts, use Minio (2020-05-01T22:19:14Z) as backend.
-
Run a CI to push images, success.
-
Try to pull images from it, failed.
Or you could just run this command:
docker pull container.anislet.dev/orwill/lighttp:latest
To pull images from my current registry and recurrent this issue.
What is the current bug behavior?
Here's what happend when I run docker pull container.anislet.dev/orwill/lighttp:latest
root@vultr:~# docker pull container.anislet.dev/orwill/lighttp
Using default tag: latest
latest: Pulling from orwill/lighttp
4f26fff4abdf: Pulling fs layer
error pulling image configuration: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>docker/registry/v2/blobs/sha256/3f/3f46894448b9c91f9d8d31b2790c9e4369bb1a874cc22cad09b0c0c369a0d74a/data</Key><BucketName>dev.anislet.code.registry</BucketName><Resource>/dev.anislet.code.registry/docker/registry/v2/blobs/sha256/3f/3f46894448b9c91f9d8d31b2790c9e4369bb1a874cc22cad09b0c0c369a0d74a/data</Resource><Region>ca-south-1</Region><RequestId>160C533EC0919A25</RequestId><HostId>86d7ab5a-9d17-4459-b888-2cd4b72b9b6c</HostId></Error>"
Here's what happend with dockerd --debug
Background
DEBU[2020-05-06T03:38:14.046766633Z] Calling HEAD /_ping
DEBU[2020-05-06T03:38:14.049744169Z] Calling POST /v1.40/images/create?fromImage=container.anislet.dev%2Forwill%2Flighttp&tag=latest
DEBU[2020-05-06T03:38:14.052505655Z] hostDir: /etc/docker/certs.d/container.anislet.dev
DEBU[2020-05-06T03:38:14.052688006Z] Trying to pull container.anislet.dev/orwill/lighttp from https://container.anislet.dev v2
DEBU[2020-05-06T03:38:16.111332984Z] Pulling ref from V2 registry: container.anislet.dev/orwill/lighttp:latest
DEBU[2020-05-06T03:38:16.115170844Z] pulling blob "sha256:4f26fff4abdfc0930df63ad53a251943c07d30e9e213a03b1dc39c5206e7c7df"
ERRO[2020-05-06T03:38:17.609226083Z] Not continuing with pull after error: error pulling image configuration: error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>docker/registry/v2/blobs/sha256/3f/3f46894448b9c91f9d8d31b2790c9e4369bb1a874cc22cad09b0c0c369a0d74a/data</Key><BucketName>dev.anislet.code.registry</BucketName><Resource>/dev.anislet.code.registry/docker/registry/v2/blobs/sha256/3f/3f46894448b9c91f9d8d31b2790c9e4369bb1a874cc22cad09b0c0c369a0d74a/data</Resource><Region>ca-south-1</Region><RequestId>160C533EC0919A25</RequestId><HostId>86d7ab5a-9d17-4459-b888-2cd4b72b9b6c</HostId></Error>"
What is the expected correct behavior?
Could pull images from my registry
Relevant logs and/or screenshots
Logs from docker side see above.
logs from registry pod:
time="2020-05-06T03:38:14.642463525Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.13.9 http.request.host=container.anislet.dev http.request.id=734ee0b1-7da7-4cb9-a2b8-9bf3ca1f4d28 http.request.method=GET http.request.remoteaddr=10.42.1.41 http.request.uri="/v2/" http.request.useragent="docker/19.03.6 go/go1.12.17 git-commit/369ce74a3c kernel/4.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.6 \(linux\))"
10.42.1.47 - - [06/May/2020:03:38:14 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/19.03.6 go/go1.12.17 git-commit/369ce74a3c kernel/4.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.6 \\(linux\\))"
10.42.1.47 - - [06/May/2020:03:38:15 +0000] "GET /v2/orwill/lighttp/manifests/latest HTTP/1.1" 200 428 "" "docker/19.03.6 go/go1.12.17 git-commit/369ce74a3c kernel/4.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.6 \\(linux\\))"
10.42.1.47 - - [06/May/2020:03:38:16 +0000] "GET /v2/orwill/lighttp/blobs/sha256:4f26fff4abdfc0930df63ad53a251943c07d30e9e213a03b1dc39c5206e7c7df HTTP/1.1" 307 0 "" "docker/19.03.6 go/go1.12.17 git-commit/369ce74a3c kernel/4.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.6 \\(linux\\))"
10.42.1.47 - - [06/May/2020:03:38:16 +0000] "GET /v2/orwill/lighttp/blobs/sha256:3f46894448b9c91f9d8d31b2790c9e4369bb1a874cc22cad09b0c0c369a0d74a HTTP/1.1" 307 0 "" "docker/19.03.6 go/go1.12.17 git-commit/369ce74a3c kernel/4.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.6 \\(linux\\))"
10.42.1.47 - - [06/May/2020:03:38:17 +0000] "GET /v2/orwill/lighttp/blobs/sha256:4f26fff4abdfc0930df63ad53a251943c07d30e9e213a03b1dc39c5206e7c7df HTTP/1.1" 307 0 "" "docker/19.03.6 go/go1.12.17 git-commit/369ce74a3c kernel/4.15.0-88-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.6 \\(linux\\))"
Results of GitLab environment info
Newest Gitlab Chart. with K3S v1.17.4+k3s1
Results of GitLab application Check
Don't know how to run this with Kubernetes.
Ps.
I believe the Advanced Troubleshooting of container registry is out of date. I tried to debug with mitmproxy fllow that guide but just can't work . Please update it asap if you could.