registry & s3: pushing fails with 'error parsing HTTP 403 response body: unexpected end of JSON input: ""'
gitlab-ce:8.9.2 docker: 1.11.2 (centos7 build) OR 1.12.0-rc2 (osx beta) - reproduced this issue on both.
We've set up gitlab's registry to use s3 as a storage backend. Whenever we try to push an image, we get an error message that looks like this:
[root@lenny tmp]# docker -D push gitlab-registry.b-lex.com/b-lex/servermanagement:webhare-ci
The push refers to a repository [gitlab-registry.b-lex.com/b-lex/servermanagement]
4fe15f8d0ae6: Pushing [==================================================>] 5.046 MB
error parsing HTTP 403 response body: unexpected end of JSON input: ""
docker login worked fine. The 403 is being sent from S3, after disabling encryption I was able to catch the request:
0x0020: 5018 00e5 c981 0000 4845 4144 202f 646f P.......HEAD./do
0x0030: 636b 6572 2f72 6567 6973 7472 792f 7632 cker/registry/v2
0x0040: 2f62 6c6f 6273 2f73 6861 3235 362f 6531 /blobs/sha256/e1
.........
0x0000: 4500 0145 ddc4 4000 3206 a2f8 36e7 82d3 E..E..@.2...6...
0x0010: 0a08 0334 0050 9d66 5d1e 9d1c 5162 565a ...4.P.f]...QbVZ
0x0020: 5018 003e febb 0000 4854 5450 2f31 2e31 P..>....HTTP/1.1
0x0030: 2034 3033 2046 6f72 6269 6464 656e 0d0a .403.Forbidden..
0x0040: 782d 616d 7a2d 7265 7175 6573 742d 6964 x-amz-request-id
The full signed URL was:
http://webhare-docker-registry.s3-eu-west-1.amazonaws.com/docker/registry/v2/blobs/sha256/e1/e110a4a1794126ef308a49f2d65785af2f25538f06700721aad8283b81fdfa58/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJY4NHZNPXN6T3YBA%2F20160629%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20160629T193758Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=f190333bcbabc644d1d06f4b13113cbf9c5543ec286443037fa79c2ae6da8827
Doing a GET on this URL works fine, so the signature is valid. However, the HTTP verb is part of a signed URL in S3, so the same URL can't work for both GET & HEAD. Apparently the registry is passing an URL expecting docker to use GET (since that's what the URL is signed for) but the docker client decides to use a HEAD request ?
This might very well be a docker upstream issue, but as I can't see anyone else reporting similar issues, and this seems pretty bad, it seemed best to check with gitlab first.