Docker push to self-hosted gitlab registry results in a 403 invalid character
Summary
Pushing a large container image layer to our self-hosted gitlab instance, fails with the following error message:
error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body>\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"
Steps to reproduce
docker pull docker:19
docker tag docker:19 <gitlab-registry>/<group>/<project>:test
docker push <gitlab-registry>/<group>/<project>:test
Example Project
In our environment, the following Dockerfile fails:
FROM alpine
RUN dd if=/dev/random of=/home/mb0 bs=1 count=$((1024*1024))
ARG layersize=13
ENV LAYER=1
RUN while [ $LAYER -le ${layersize} ]; do cp /home/mb0 /home/mb$LAYER; LAYER=$(($LAYER+1)); done
With --build-arg layersize 12
the image can be pushed.
What is the current bug behavior?
Docker image layer push fails with following error at the end of the upload of the layer:
error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body>\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"
What is the expected correct behavior?
Docker is able to successfully push the layer.
Relevant logs and/or screenshots
docker push gitlab-registry.binx.io/playground/app:test
The push refers to repository [gitlab-registry.binx.io/playground/app:test]
b35868ff34af: Pushing [==================================================>] 13.64MB
b21ca43130e1: Layer already exists
50644c29ef5a: Layer already exists
error parsing HTTP 403 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body>\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n"
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Possible fixes
n/a
forum post
Edited by Jason Plum