registry & s3: pushing fails with 'error parsing HTTP 403 response body: unexpected end of JSON input: ""'
gitlab-ce:8.9.2 docker: 1.11.2 (centos7 build) OR 1.12.0-rc2 (osx beta) - reproduced this issue on both.
We've set up gitlab's registry to use s3 as a storage backend. Whenever we try to push an image, we get an error message that looks like this:
[root@lenny tmp]# docker -D push gitlab-registry.b-lex.com/b-lex/servermanagement:webhare-ci The push refers to a repository [gitlab-registry.b-lex.com/b-lex/servermanagement] 4fe15f8d0ae6: Pushing [==================================================>] 5.046 MB error parsing HTTP 403 response body: unexpected end of JSON input: ""
docker login worked fine. The 403 is being sent from S3, after disabling encryption I was able to catch the request:
0x0020: 5018 00e5 c981 0000 4845 4144 202f 646f P.......HEAD./do 0x0030: 636b 6572 2f72 6567 6973 7472 792f 7632 cker/registry/v2 0x0040: 2f62 6c6f 6273 2f73 6861 3235 362f 6531 /blobs/sha256/e1 ......... 0x0000: 4500 0145 ddc4 4000 3206 a2f8 36e7 82d3 E..E..@.2...6... 0x0010: 0a08 0334 0050 9d66 5d1e 9d1c 5162 565a ...4.P.f]...QbVZ 0x0020: 5018 003e febb 0000 4854 5450 2f31 2e31 P..>....HTTP/1.1 0x0030: 2034 3033 2046 6f72 6269 6464 656e 0d0a .403.Forbidden.. 0x0040: 782d 616d 7a2d 7265 7175 6573 742d 6964 x-amz-request-id
The full signed URL was:
Doing a GET on this URL works fine, so the signature is valid. However, the HTTP verb is part of a signed URL in S3, so the same URL can't work for both GET & HEAD. Apparently the registry is passing an URL expecting docker to use GET (since that's what the URL is signed for) but the docker client decides to use a HEAD request ?
This might very well be a docker upstream issue, but as I can't see anyone else reporting similar issues, and this seems pretty bad, it seemed best to check with gitlab first.