Registry behind Nginx - Error: writing blob: uploading layer chunked: StatusCode: 404
I have a problem when I want to upload larger images to the GitLab 15.2.1-ee omnibus registry. Smaller images are working fine.
$ podman --log-level=debug push gitlab.foo.eu/fred/foo:latest
[...]
Copying blob 19c87d469c81 skipped: already exists
Copying blob 636dfc49dd62 skipped: already exists
Copying blob ec4972933382 skipped: already exists
Copying blob 99ccac390b0d skipped: already exists
Copying blob 99b8bdec6ac3 skipped: already exists
Copying blob f31f7d599270 skipped: already exists
Copying blob 0ac2f57de551 [===========================>----------] 7.5GiB / 10.2GiB
Copying blob 217f9a760626 skipped: already exists
Copying blob 1af5510e706f skipped: already exists
DEBU[0081] Looking up image "gitlab.foo.eu/fred/foo:latest" in local containers storage
DEBU[0081] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0081] Trying "gitlab.foo.eu/fred/foo:latest" ...
DEBU[0081] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@f3499860a14fed43b4f6c62ba932a646dc7e8c6fc05b7a43a3235cc65637506c"
DEBU[0081] Found image "gitlab.foo.eu/fred/foo:latest" as "gitlab.foo.eu/fred/foo:latest" in local containers storage
Error: writing blob: uploading layer chunked: StatusCode: 404, <html>
...ad><title>404 Not Found</title></head>
gitlab.rb
registry_external_url 'https://gitlab.foo.eu'
registry['env'] = {
"REGISTRY_HTTP_RELATIVEURLS" => true
}
registry_nginx['listen_addresses'] = ['*', '[::]']
registry_nginx['listen_port'] = 5050
registry_nginx['listen_https'] = false
nginx:
location /v2/ {
# Do not allow connections from docker 1.5 and earlier
# docker pre-1.6.0 did not properly set the user agent on ping, catch "Go *" user agents
if ($http_user_agent ~ "^(docker\/1\.(3|4|5(?!\.[0-9]-dev))|Go ).*$" ) {
return 404;
}
# required to avoid HTTP 411: see Issue #1486 (https://github.com/moby/moby/issues/1486)
chunked_transfer_encoding on;
# disable any limits to avoid HTTP 413 for large image uploads
client_max_body_size 0;
## Individual nginx logs for this vhost
access_log /var/log/nginx/container_registry_ssl_access.log;
error_log /var/log/nginx/container_registry_ssl_error.log;
## If $docker_distribution_api_version is empty, the header is not added.
## See the map directive above where this variable is defined.
add_header 'Docker-Distribution-Api-Version' $docker_distribution_api_version always;
proxy_set_header Host $http_host; # required for docker client's sake
proxy_set_header X-Real-IP $remote_addr; # pass on real client's IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;
proxy_read_timeout 900;
### do not fill up '/var' before handing uploads down to the GitLab's container-registry-nginx.
proxy_max_temp_file_size 0;
proxy_cache off;
proxy_buffering off;
proxy_request_buffering off;
proxy_pass http://container_registry;
}