Dependency Proxy uses wrong Content-Type
Summary
Dependency Proxy uses Content-Type: application/json
instead of any of the types expected by go-containerregistry (which is used by kaniko):
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Crane is strict about the content type by design.
Steps to reproduce
- Enable the dependency proxy
- Create a personal access token with the scope "api"
- Run the go-containerregistry debug image:
docker run --rm -it --entrypoint "/busybox/sh" gcr.io/go-containerregistry/crane:debug
- Log in:
crane auth login -u "<username>" -p "<token>" gitlab.example.com
- Try to pull image via your dependency proxy:
crane -v pull gitlab.example.com/some-group/dependency_proxy/containers/redis:6.0.10 blub.tar
Example Project
https://gitlab.com/debug-kaniko/debug-kaniko-error
The behaviour on Gitlab.com is slightly different especially for Kaniko but crane shows the exact same error as with an on-premise installation at line 267 of the job log.
What is the current bug behavior?
The Gitlab server's response containing the container manifest uses the Content-Type "application/json".
What is the expected correct behavior?
The Gitlab server replies using the appropriate Content-Type out of this list:
Accept:
application/vnd.docker.distribution.manifest.v1+json,
application/vnd.docker.distribution.manifest.v1+prettyjws,
application/vnd.docker.distribution.manifest.v2+json,
application/vnd.oci.image.manifest.v1+json,
application/vnd.docker.distribution.manifest.list.v2+json,
application/vnd.oci.image.index.v1+json
Relevant logs and/or screenshots
crane debug log
/ # crane -v pull gitlab.example.com/some-group/dependency_proxy/containers/redis:6.0.10 test.tar 2021/01/22 16:17:12 --> GET https://gitlab.example.com/v2/ 2021/01/22 16:17:12 GET /v2/ HTTP/1.1 Host: gitlab.example.com User-Agent: crane/859dff29bf3017e5bd7f0bdd29d716913c1131b0 go-containerregistry Accept-Encoding: gzip2021/01/22 16:17:12 <-- 401 https://gitlab.example.com/v2/ (196.305506ms) 2021/01/22 16:17:12 HTTP/2.0 401 Unauthorized Cache-Control: no-cache Content-Type: text/plain; charset=utf-8 Date: Fri, 22 Jan 2021 16:17:12 GMT Server: nginx Www-Authenticate: Bearer realm="https://gitlab.example.com/jwt/auth",service="dependency_proxy" X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: SAMEORIGIN X-Permitted-Cross-Domain-Policies: none X-Request-Id: 01EWNE59PYRSWYM4DZW1S1N0TA X-Runtime: 0.035039 X-Xss-Protection: 1; mode=block Content-Length: 0
2021/01/22 16:17:12 --> GET https://gitlab.example.com/jwt/auth?scope=repository%3Asome-group%2Fdependency_proxy%2Fcontainers%2Fredis%3Apull&service=dependency_proxy [body redacted: basic token response contains credentials] 2021/01/22 16:17:12 GET /jwt/auth?scope=repository%3Asome-group%2Fdependency_proxy%2Fcontainers%2Fredis%3Apull&service=dependency_proxy HTTP/1.1 Host: gitlab.example.com User-Agent: crane/859dff29bf3017e5bd7f0bdd29d716913c1131b0 go-containerregistry Authorization: Accept-Encoding: gzip
2021/01/22 16:17:13 <-- 200 https://gitlab.example.com/jwt/auth?scope=repository%3Asome-group%2Fdependency_proxy%2Fcontainers%2Fredis%3Apull&service=dependency_proxy (82.793499ms) [body redacted: basic token response contains credentials] 2021/01/22 16:17:13 HTTP/2.0 200 OK Content-Length: 240 Cache-Control: max-age=0, private, must-revalidate Content-Type: application/json; charset=utf-8 Date: Fri, 22 Jan 2021 16:17:13 GMT Etag: W/"96cb59aacb118e76873f41ea9ecf4320" Referrer-Policy: strict-origin-when-cross-origin Server: nginx Set-Cookie: experimentation_subject_id=redacted; path=/; expires=Tue, 22 Jan 2041 16:17:13 GMT; secure; HttpOnly; SameSite=None Strict-Transport-Security: max-age=31536000 X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: DENY X-Gitlab-Feature-Category: authentication_and_authorization X-Permitted-Cross-Domain-Policies: none X-Request-Id: 01EWNE59S4F6JNCP2QKNVAM35R X-Runtime: 0.047994 X-Ua-Compatible: IE=edge X-Xss-Protection: 1; mode=block
2021/01/22 16:17:13 --> GET https://gitlab.example.com/v2/some-group/dependency_proxy/containers/redis/manifests/6.0.10 2021/01/22 16:17:13 GET /v2/some-group/dependency_proxy/containers/redis/manifests/6.0.10 HTTP/1.1 Host: gitlab.example.com User-Agent: crane/859dff29bf3017e5bd7f0bdd29d716913c1131b0 go-containerregistry Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json Authorization: Accept-Encoding: gzip
2021/01/22 16:17:14 <-- 200 https://gitlab.example.com/v2/some-group/dependency_proxy/containers/redis/manifests/6.0.10 (1.079278763s) 2021/01/22 16:17:14 HTTP/2.0 200 OK Cache-Control: max-age=0, private, must-revalidate, no-store Content-Disposition: attachment; filename="redis%3A6.0.10.json"; filename*=UTF-8''redis%3A6.0.10.json Content-Transfer-Encoding: binary Content-Type: application/json Date: Fri, 22 Jan 2021 16:17:14 GMT Last-Modified: Fri, 22 Jan 2021 13:28:58 GMT Pragma: no-cache Referrer-Policy: strict-origin-when-cross-origin Server: nginx Set-Cookie: experimentation_subject_id=; path=/; expires=Tue, 22 Jan 2041 16:17:13 GMT; secure; HttpOnly; SameSite=None Set-Cookie: _gitlab_session=redacted; path=/; secure; HttpOnly; SameSite=None Strict-Transport-Security: max-age=31536000 Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: DENY X-Gitlab-Feature-Category: dependency_proxy X-Permitted-Cross-Domain-Policies: none X-Request-Id: 01EWNE59V39S1JKFKTAEWZC8EJ X-Runtime: 1.064720 X-Ua-Compatible: IE=edge X-Xss-Protection: 1; mode=block
{ "schemaVersion": 1, "name": "library/redis", "tag": "6.0.10", "architecture": "amd64", "fsLayers": [ { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:3f703e7f380f529a9928929c9341638088bf12a1ff7d186f9973b36a7832db09" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:56b9a72e68ff3f982b2d7538decfdb234380decb6fa657d0836adada6b496507" }, { "blobSum": "sha256:ee99c35818f884815f57a706b4ca2b4a6b70bfebc032579ffef52e8eb6a1ac5f" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:ce21c8a3dbee27bfdb58dfcfbd5b0a7186659ca37edfae38a590fc37c90089a2" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:f40dd07fe7be6f8bcbaaf467cd650465af6c33f0d29615a4b2838450d3db6be8" }, { "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" }, { "blobSum": "sha256:a076a628af6f7dcabc536bee373c0d9b48d9f0516788e64080c4e841746e6ce6" } ], "history": [ { "v1Compatibility": "{"architecture":"amd64","config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"6379/tcp":{}},"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","GOSU_VERSION=1.12","REDIS_VERSION=6.0.10","REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-6.0.10.tar.gz","REDIS_DOWNLOAD_SHA=79bbb894f9dceb33ca699ee3ca4a4e1228be7fb5547aeb2f99d921e86c1285bd"],"Cmd":["redis-server"],"Image":"sha256:222c0cecc006d8c73a04a58b5fa15ebae171a6e82a8ee8650ae616f6f1798ef4","Volumes":{"/data":{}},"WorkingDir":"/data","Entrypoint":["docker-entrypoint.sh"],"OnBuild":null,"Labels":null},"container":"16535cfaf84a4049b6c02840219e8473787d5610e29409049df3a41bbf77a333","container_config":{"Hostname":"16535cfaf84a","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"ExposedPorts":{"6379/tcp":{}},"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","GOSU_VERSION=1.12","REDIS_VERSION=6.0.10","REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-6.0.10.tar.gz","REDIS_DOWNLOAD_SHA=79bbb894f9dceb33ca699ee3ca4a4e1228be7fb5547aeb2f99d921e86c1285bd"],"Cmd":["/bin/sh","-c","#(nop) ","CMD [\"redis-server\"]"],"Image":"sha256:222c0cecc006d8c73a04a58b5fa15ebae171a6e82a8ee8650ae616f6f1798ef4","Volumes":{"/data":{}},"WorkingDir":"/data","Entrypoint":["docker-entrypoint.sh"],"OnBuild":null,"Labels":{}},"created":"2021-01-13T09:45:41.527587343Z","docker_version":"19.03.12","id":"2ca0f04db0234855db181ab91e0ef5f350df51f44c44d13f463c4e8315341016","os":"linux","parent":"520f0ec588b3eb19c93dd61d10e36033c257c36f059588bfd3be32e446280831","throwaway":true}" }, { "v1Compatibility": "{"id":"520f0ec588b3eb19c93dd61d10e36033c257c36f059588bfd3be32e446280831","parent":"361ebcb32ff845d0ee5cc15d37fcb1e3af229dc9f1f540a07bdd6f70fdae13ac","created":"2021-01-13T09:45:41.348366093Z","container_config":{"Cmd":["/bin/sh -c #(nop) EXPOSE 6379"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"361ebcb32ff845d0ee5cc15d37fcb1e3af229dc9f1f540a07bdd6f70fdae13ac","parent":"ff34d98f211685aa00911a5291c7714472c4fdddde1604b756b1306401864252","created":"2021-01-13T09:45:41.168080832Z","container_config":{"Cmd":["/bin/sh -c #(nop) ENTRYPOINT [\"docker-entrypoint.sh\"]"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"ff34d98f211685aa00911a5291c7714472c4fdddde1604b756b1306401864252","parent":"66cba2d45512db49f459b9fc7caa7464cea38746d79b045e9c63db0f07616f6c","created":"2021-01-13T09:45:41.004454972Z","container_config":{"Cmd":["/bin/sh -c #(nop) COPY file:df205a0ef6e6df8947ce0a7ae9e37b6a5588035647f38a49b8b07321003a8a01 in /usr/local/bin/ "]}}" }, { "v1Compatibility": "{"id":"66cba2d45512db49f459b9fc7caa7464cea38746d79b045e9c63db0f07616f6c","parent":"1ba39a16e8ac9df47ad095b1f4389e55b74f01baec1bb7d5ecbb0bb9fda16ceb","created":"2021-01-13T09:45:40.759518089Z","container_config":{"Cmd":["/bin/sh -c #(nop) WORKDIR /data"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"1ba39a16e8ac9df47ad095b1f4389e55b74f01baec1bb7d5ecbb0bb9fda16ceb","parent":"6dbada199b38dd91c78187817d25c3e0ecb3b387d18248237290c4a0534f864f","created":"2021-01-13T09:45:40.520238281Z","container_config":{"Cmd":["/bin/sh -c #(nop) VOLUME [/data]"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"6dbada199b38dd91c78187817d25c3e0ecb3b387d18248237290c4a0534f864f","parent":"b52acb0017d316b960a9a1fa80c11de8d5d1df9b23ad9cfee8b56706f3239323","created":"2021-01-13T09:45:40.355349666Z","container_config":{"Cmd":["/bin/sh -c mkdir /data \u0026\u0026 chown redis:redis /data"]}}" }, { "v1Compatibility": "{"id":"b52acb0017d316b960a9a1fa80c11de8d5d1df9b23ad9cfee8b56706f3239323","parent":"ec8254d71597e1442d4f17810ca9bff1d2ed31056fa1784ae1aeee04b6a70092","created":"2021-01-13T09:45:39.387835221Z","container_config":{"Cmd":["/bin/sh -c set -eux; \t\tsavedAptMark=\"$(apt-mark showmanual)\"; \tapt-get update; \tapt-get install -y --no-install-recommends \t\tca-certificates \t\twget \t\t\t\tdpkg-dev \t\tgcc \t\tlibc6-dev \t\tlibssl-dev \t\tmake \t; \trm -rf /var/lib/apt/lists/; \t\twget -O redis.tar.gz \"$REDIS_DOWNLOAD_URL\"; \techo \"$REDIS_DOWNLOAD_SHA redis.tar.gz\" | sha256sum -c -; \tmkdir -p /usr/src/redis; \ttar -xzf redis.tar.gz -C /usr/src/redis --strip-components=1; \trm redis.tar.gz; \t\tgrep -E '^ createBoolConfig[(]\"protected-mode\",., 1 ,.[)],$' /usr/src/redis/src/config.c; \tsed -ri 's!^( createBoolConfig[(]\"protected-mode\",., )1( ,.[)],)$!\\10\\2!' /usr/src/redis/src/config.c; \tgrep -E '^ createBoolConfig[(]\"protected-mode\",., 0 ,.[)],$' /usr/src/redis/src/config.c; \t\tgnuArch=\"$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)\"; \textraJemallocConfigureFlags=\"--build=$gnuArch\"; \tdpkgArch=\"$(dpkg --print-architecture)\"; \tcase \"${dpkgArch##-}\" in \t\tamd64 | i386 | x32) extraJemallocConfigureFlags=\"$extraJemallocConfigureFlags --with-lg-page=12\" ;; \t\t) extraJemallocConfigureFlags=\"$extraJemallocConfigureFlags --with-lg-page=16\" ;; \tesac; \textraJemallocConfigureFlags=\"$extraJemallocConfigureFlags --with-lg-hugepage=21\"; \tgrep -F 'cd jemalloc \u0026\u0026 ./configure ' /usr/src/redis/deps/Makefile; \tsed -ri 's!cd jemalloc \u0026\u0026 ./configure !\u0026'\"$extraJemallocConfigureFlags\"' !' /usr/src/redis/deps/Makefile; \tgrep -F \"cd jemalloc \u0026\u0026 ./configure $extraJemallocConfigureFlags \" /usr/src/redis/deps/Makefile; \t\texport BUILD_TLS=yes; \tmake -C /usr/src/redis -j \"$(nproc)\" all; \tmake -C /usr/src/redis install; \t\tserverMd5=\"$(md5sum /usr/local/bin/redis-server | cut -d' ' -f1)\"; export serverMd5; \tfind /usr/local/bin/redis -maxdepth 0 \t\t-type f -not -name redis-server \t\t-exec sh -eux -c ' \t\t\tmd5=\"$(md5sum \"$1\" | cut -d\" \" -f1)\"; \t\t\ttest \"$md5\" = \"$serverMd5\"; \t\t' -- '{}' ';' \t\t-exec ln -svfT 'redis-server' '{}' ';' \t; \t\trm -r /usr/src/redis; \t\tapt-mark auto '.' \u003e /dev/null; \t[ -z \"$savedAptMark\" ] || apt-mark manual $savedAptMark \u003e /dev/null; \tfind /usr/local -type f -executable -exec ldd '{}' ';' \t\t| awk '/=\u003e/ { print $(NF-1) }' \t\t| sort -u \t\t| xargs -r dpkg-query --search \t\t| cut -d: -f1 \t\t| sort -u \t\t| xargs -r apt-mark manual \t; \tapt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \t\tredis-cli --version; \tredis-server --version"]}}" }, { "v1Compatibility": "{"id":"ec8254d71597e1442d4f17810ca9bff1d2ed31056fa1784ae1aeee04b6a70092","parent":"e0b8851b4cac0cedc900d4a1d52b46199587ddfa050afceeed721f9869901490","created":"2021-01-13T09:44:39.874761475Z","container_config":{"Cmd":["/bin/sh -c #(nop) ENV REDIS_DOWNLOAD_SHA=79bbb894f9dceb33ca699ee3ca4a4e1228be7fb5547aeb2f99d921e86c1285bd"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"e0b8851b4cac0cedc900d4a1d52b46199587ddfa050afceeed721f9869901490","parent":"375ecb2ca7c121f6504eb88e7245b1f37e12890eb1c628646c8a98f5a58933e3","created":"2021-01-13T09:44:39.695211643Z","container_config":{"Cmd":["/bin/sh -c #(nop) ENV REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-6.0.10.tar.gz"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"375ecb2ca7c121f6504eb88e7245b1f37e12890eb1c628646c8a98f5a58933e3","parent":"2b58f68944e520a14be395ec2f132dd6b2c629f091572db447e20755fb7a3a7e","created":"2021-01-13T09:44:39.486592532Z","container_config":{"Cmd":["/bin/sh -c #(nop) ENV REDIS_VERSION=6.0.10"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"2b58f68944e520a14be395ec2f132dd6b2c629f091572db447e20755fb7a3a7e","parent":"760d597e2867ff4a5a2ec84a7159c6a5a1104dbc1c00ae1e3d1475bb5b397c0d","created":"2021-01-12T18:54:13.74212158Z","container_config":{"Cmd":["/bin/sh -c set -eux; \tsavedAptMark=\"$(apt-mark showmanual)\"; \tapt-get update; \tapt-get install -y --no-install-recommends ca-certificates dirmngr gnupg wget; \trm -rf /var/lib/apt/lists/; \tdpkgArch=\"$(dpkg --print-architecture | awk -F- '{ print $NF }')\"; \twget -O /usr/local/bin/gosu \"https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch\"; \twget -O /usr/local/bin/gosu.asc \"https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch.asc\"; \texport GNUPGHOME=\"$(mktemp -d)\"; \tgpg --batch --keyserver hkps://keys.openpgp.org --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4; \tgpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu; \tgpgconf --kill all; \trm -rf \"$GNUPGHOME\" /usr/local/bin/gosu.asc; \tapt-mark auto '.*' \u003e /dev/null; \t[ -z \"$savedAptMark\" ] || apt-mark manual $savedAptMark \u003e /dev/null; \tapt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \tchmod +x /usr/local/bin/gosu; \tgosu --version; \tgosu nobody true"]}}" }, { "v1Compatibility": "{"id":"760d597e2867ff4a5a2ec84a7159c6a5a1104dbc1c00ae1e3d1475bb5b397c0d","parent":"1166e1457c1804b4f00495f395da9120b24da16636897db2d202b144277a33ea","created":"2021-01-12T18:53:57.021544068Z","container_config":{"Cmd":["/bin/sh -c #(nop) ENV GOSU_VERSION=1.12"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"1166e1457c1804b4f00495f395da9120b24da16636897db2d202b144277a33ea","parent":"663d15fd147c98a9f8e313c10a15a3da126b9bfc12e410747056be6c48183c54","created":"2021-01-12T18:53:56.704276999Z","container_config":{"Cmd":["/bin/sh -c groupadd -r -g 999 redis \u0026\u0026 useradd -r -g redis -u 999 redis"]}}" }, { "v1Compatibility": "{"id":"663d15fd147c98a9f8e313c10a15a3da126b9bfc12e410747056be6c48183c54","parent":"06ca0703b4a50fa19d36a98c0c3335f3260d4e6854ebe4294122d4cc85216c8c","created":"2021-01-12T00:32:51.939863901Z","container_config":{"Cmd":["/bin/sh -c #(nop) CMD [\"bash\"]"]},"throwaway":true}" }, { "v1Compatibility": "{"id":"06ca0703b4a50fa19d36a98c0c3335f3260d4e6854ebe4294122d4cc85216c8c","created":"2021-01-12T00:32:51.692277281Z","container_config":{"Cmd":["/bin/sh -c #(nop) ADD file:422aca8901ae3d869a815051cea7f1e4c0204fad16884e7cd01da57d142f2e3a in / "]}}" } ], "signatures": [ { "header": { "jwk": { "crv": "P-256", "kid": "BAKB:OWRO:ZISI:KA6F:2DY2:TG5V:NMQD:JOK5:HGVB:IL6H:SSSQ:MFSH", "kty": "EC", "x": "-IYlGFI1Td9f9pkzBaaMq8H0O_sC78JBLPg0t89rEMA", "y": "B-oP20FAArmtK-cwCNheaqzxdVIQ9av9kiS7npeHWiE" }, "alg": "ES256" }, "signature": "JILX2y0x1t6-gzhdEIBBpqyfta4_u7aBqQSeO9cM_zEkljdBiowy3UTzh709Jdsd50viJk7MM_qgdoA41q4hcg", "protected": "eyJmb3JtYXRMZW5ndGgiOjEzMDA1LCJmb3JtYXRUYWlsIjoiQ24wIiwidGltZSI6IjIwMjEtMDEtMjJUMTM6Mjg6NThaIn0" } ] } 2021/01/22 16:17:14 Unexpected media type for Image(): application/json 2021/01/22 16:17:14 --> GET https://gitlab.example.com/v2/some-group/dependency_proxy/containers/redis/blobs/: 2021/01/22 16:17:14 GET /v2/some-group/dependency_proxy/containers/redis/blobs/: HTTP/1.1 Host: gitlab.example.com User-Agent: crane/859dff29bf3017e5bd7f0bdd29d716913c1131b0 go-containerregistry Authorization: Accept-Encoding: gzip
2021/01/22 16:17:14 <-- 302 https://gitlab.example.com/v2/some-group/dependency_proxy/containers/redis/blobs/: (118.984792ms) 2021/01/22 16:17:14 HTTP/2.0 302 Found Content-Length: 99 Cache-Control: no-cache Content-Type: text/html; charset=utf-8 Date: Fri, 22 Jan 2021 16:17:14 GMT Location: https://gitlab.example.com/users/sign_in Referrer-Policy: strict-origin-when-cross-origin Server: nginx Set-Cookie: experimentation_subject_id=; path=/; expires=Tue, 22 Jan 2041 16:17:14 GMT; secure; HttpOnly; SameSite=None Set-Cookie: _gitlab_session=83cfb675c7deccab486656ccc3d97a39; path=/; expires=Fri, 22 Jan 2021 18:17:14 GMT; secure; HttpOnly; SameSite=None Strict-Transport-Security: max-age=31536000 X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: DENY X-Gitlab-Feature-Category: X-Permitted-Cross-Domain-Policies: none X-Request-Id: 01EWNE5AZDXPP673TADADJ1DSX X-Runtime: 0.025020 X-Ua-Compatible: IE=edge X-Xss-Protection: 1; mode=block
You are being redirected. 2021/01/22 16:17:14 --> GET https://gitlab.example.com/users/sign_in 2021/01/22 16:17:14 GET /users/sign_in HTTP/1.1 Host: gitlab.example.com User-Agent: crane/859dff29bf3017e5bd7f0bdd29d716913c1131b0 go-containerregistry Authorization: Referer: https://gitlab.example.com/v2/some-group/dependency_proxy/containers/redis/blobs/: Accept-Encoding: gzip2021/01/22 16:17:14 <-- 200 https://gitlab.example.com/users/sign_in (142.850875ms) 2021/01/22 16:17:14 HTTP/2.0 200 OK Cache-Control: max-age=0, private, must-revalidate Content-Type: text/html; charset=utf-8 Date: Fri, 22 Jan 2021 16:17:14 GMT Etag: W/"6316dcb6246617c1d3ef99480476d4a6" Referrer-Policy: strict-origin-when-cross-origin Server: nginx Set-Cookie: experimentation_subject_id=; path=/; expires=Tue, 22 Jan 2041 16:17:14 GMT; secure; HttpOnly; SameSite=None Set-Cookie: _gitlab_session=redacted; path=/; expires=Fri, 22 Jan 2021 18:17:14 GMT; secure; HttpOnly; SameSite=None Strict-Transport-Security: max-age=31536000 Vary: Accept-Encoding X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: DENY X-Gitlab-Feature-Category: authentication_and_authorization X-Permitted-Cross-Domain-Policies: none X-Request-Id: 01EWNE5B2PE9XC9Z93NA195P19 X-Runtime: 0.060487 X-Ua-Compatible: IE=edge X-Xss-Protection: 1; mode=block
####### Response body is the HTML login page - removed #######
2021/01/22 16:17:14 saving tarball test.tar: unable to calculate manifest: unsupported hash: ""
Kaniko error
DEBU[0000] Copying file /builds/some-group/some-project/Dockerfile to /kaniko/Dockerfile TRAC[0000] Adding /var/run to initialIgnoreList DEBU[0000] Skip resolving path /kaniko/Dockerfile DEBU[0000] Skip resolving path /builds/some-group/some-project/ DEBU[0000] Skip resolving path /cache DEBU[0000] Skip resolving path DEBU[0000] Skip resolving path DEBU[0000] Skip resolving path DEBU[0000] Built stage name to index map: map[] INFO[0000] Retrieving image manifest gitlab.example.com:443/some-group/dependency_proxy/containers/redis:6.0.10 INFO[0000] Retrieving image gitlab.example.com:443/some-group/dependency_proxy/containers/redis:6.0.10 DEBU[0001] No file found for cache key sha256:310eb6c69489d450517aa8f5bde2bd71145ae544b48e217cae6739ece5306783 stat /cache/sha256:310eb6c69489d450517aa8f5bde2bd71145ae544b48e217cae6739ece5306783: no such file or directory DEBU[0001] Image gitlab.example.com:443/some-group/dependency_proxy/containers/redis:6.0.10 not found in cache INFO[0001] Retrieving image manifest gitlab.example.com:443/some-group/dependency_proxy/containers/redis:6.0.10 INFO[0001] Retrieving image gitlab.example.com:443/some-group/dependency_proxy/containers/redis:6.0.10 error building image: unsupported hash: ""
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: CentOS 7.9.2009 Proxy: no Current User: git Using RVM: no Ruby Version: 2.7.2p137 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.1 Redis Version: 5.0.9 Git Version: 2.29.0 Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.7.4-ee Revision: 368b4fb2eee Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.4 URL: https:// HTTP Clone URL: https:///some-group/some-project.git SSH Clone URL: git@:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: yes Using Omniauth: yes Omniauth Providers: GitLab Shell Version: 13.14.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.14.0 ? ... OK (13.14.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 7/1 ... yes 7/2 ... yes 7/4 ... yes 9/10 ... yes 7/11 ... yes 7/12 ... yes 63/13 ... yes 15/14 ... yes 15/15 ... yes 18/16 ... yes 18/17 ... yes 7/18 ... yes 7/19 ... yes 15/20 ... yes 22/22 ... yes 22/23 ... yes 24/24 ... yes 15/25 ... yes 7/27 ... yes 7/28 ... yes 7/30 ... yes 7/31 ... yes 7/33 ... yes 40/34 ... yes 7/35 ... yes 15/36 ... yes 15/37 ... yes 7/39 ... yes 24/40 ... yes 24/49 ... yes 93/54 ... yes 93/56 ... yes 93/57 ... yes 93/58 ... yes 40/60 ... yes 40/61 ... yes 24/62 ... yes 93/64 ... yes 93/65 ... yes 93/66 ... yes 93/67 ... yes 40/68 ... yes 40/69 ... yes 24/70 ... yes 15/72 ... yes 93/73 ... yes 30/74 ... yes 93/75 ... yes 40/76 ... yes 40/77 ... yes 24/80 ... yes 24/81 ... yes 27/82 ... yes 39/84 ... yes 33/85 ... yes 40/86 ... yes 36/88 ... yes 33/89 ... yes 33/90 ... yes 33/91 ... yes 33/92 ... yes 33/93 ... yes 33/94 ... yes 33/95 ... yes 39/96 ... yes 40/98 ... yes 24/99 ... yes 40/100 ... yes 28/101 ... yes 28/102 ... yes 27/103 ... yes 24/107 ... yes 39/109 ... yes 28/110 ... yes 40/113 ... yes 40/114 ... yes 40/115 ... yes 40/116 ... yes 40/117 ... yes 40/118 ... yes 40/119 ... yes 40/120 ... yes 40/121 ... yes 40/123 ... yes 40/124 ... yes 40/125 ... yes 40/126 ... yes 40/127 ... yes 28/128 ... yes 28/129 ... yes 28/130 ... yes 28/131 ... yes 40/132 ... yes 41/133 ... yes 41/134 ... yes 41/135 ... yes 41/136 ... yes 41/137 ... yes 41/138 ... yes 28/139 ... yes 22/140 ... yes 22/142 ... yes 40/143 ... yes 40/144 ... yes 40/145 ... yes 40/146 ... yes 22/147 ... yes 28/148 ... yes 24/149 ... yes 40/150 ... yes 22/151 ... yes 22/152 ... yes 22/153 ... yes 22/154 ... yes 15/156 ... yes 22/159 ... yes 22/160 ... yes 22/161 ... yes 40/162 ... yes 28/165 ... yes 7/167 ... yes 33/168 ... yes 93/169 ... yes 28/170 ... yes 28/171 ... yes 93/172 ... yes 93/173 ... yes 93/174 ... yes 24/175 ... yes 39/176 ... yes 28/177 ... yes 39/178 ... yes 39/179 ... yes 39/180 ... yes 39/181 ... yes 28/182 ... yes 27/192 ... yes 48/193 ... yes 48/194 ... yes 48/195 ... yes 48/196 ... yes 48/197 ... yes 48/198 ... yes 48/199 ... yes 48/200 ... yes 48/201 ... yes 39/202 ... yes 39/203 ... yes 39/204 ... yes 48/205 ... yes 49/206 ... yes 50/207 ... yes 28/208 ... yes 50/209 ... yes 93/210 ... yes 42/211 ... yes 28/213 ... yes 39/214 ... yes 33/215 ... yes 33/216 ... yes 33/218 ... yes 54/219 ... yes 30/221 ... yes 41/222 ... yes 41/223 ... yes 41/224 ... yes 93/225 ... yes 46/227 ... yes 48/228 ... yes 50/233 ... yes 47/234 ... yes 62/235 ... yes 62/236 ... yes 48/237 ... yes 48/238 ... yes 48/239 ... yes 48/240 ... yes 93/241 ... yes 33/245 ... yes 46/246 ... yes 63/247 ... yes 65/248 ... yes 7/250 ... yes 65/251 ... yes 65/252 ... yes 63/253 ... yes 63/254 ... yes 66/255 ... yes 93/256 ... yes 24/260 ... yes 93/261 ... yes 65/262 ... yes 67/268 ... yes 65/269 ... yes 66/270 ... yes 33/271 ... yes 71/279 ... yes 72/280 ... yes 72/281 ... yes 72/282 ... yes 72/283 ... yes 73/284 ... yes 73/285 ... yes 73/286 ... yes 74/287 ... yes 74/288 ... yes 77/289 ... yes 77/291 ... yes 93/292 ... yes 80/293 ... yes 80/294 ... yes 80/295 ... yes 39/296 ... yes 65/297 ... yes 24/299 ... yes 93/300 ... yes 22/302 ... yes 65/303 ... yes 65/304 ... yes 24/306 ... yes 93/308 ... yes 93/310 ... yes 93/313 ... yes 23/314 ... yes 48/316 ... yes 93/318 ... yes 48/319 ... yes 24/320 ... yes 85/321 ... yes 85/322 ... yes 15/323 ... yes 7/324 ... yes 24/326 ... yes 93/327 ... yes 93/329 ... yes 90/332 ... yes 49/333 ... yes 93/334 ... yes 63/335 ... yes 91/336 ... yes 91/337 ... yes 91/338 ... yes 28/339 ... yes 63/341 ... yes 93/343 ... yes 48/349 ... yes 28/353 ... yes 39/354 ... yes 63/356 ... yes 63/357 ... yes 40/358 ... yes 50/359 ... yes 93/360 ... yes 68/362 ... yes 91/363 ... yes 63/364 ... yes 63/365 ... yes 63/366 ... yes 63/368 ... yes 63/369 ... yes 63/370 ... yes 63/371 ... yes 63/372 ... yes 63/374 ... yes 63/375 ... yes 63/376 ... yes 63/377 ... yes 63/378 ... yes 63/379 ... yes 65/380 ... yes 63/381 ... yes 40/383 ... yes 40/384 ... yes 40/385 ... yes 40/386 ... yes 40/387 ... yes 40/388 ... yes 40/389 ... yes 40/390 ... yes 48/391 ... yes 36/392 ... yes 36/393 ... yes 99/394 ... yes 100/399 ... yes 46/418 ... yes 39/419 ... yes 100/420 ... yes 111/422 ... yes 113/425 ... yes 113/426 ... yes 113/427 ... yes 114/428 ... yes 114/429 ... yes 114/430 ... yes 115/431 ... yes 116/432 ... yes 116/433 ... yes 117/435 ... yes 117/436 ... yes 117/437 ... yes 46/438 ... yes 46/439 ... yes 118/440 ... yes 120/442 ... yes 120/443 ... yes 120/444 ... yes 111/445 ... yes 93/446 ... yes 48/447 ... yes 63/450 ... yes 111/451 ... yes 63/452 ... yes 63/453 ... yes 63/454 ... yes 63/456 ... yes 63/457 ... yes 46/458 ... yes 63/459 ... yes 63/460 ... yes 63/461 ... yes 63/462 ... yes 93/463 ... yes 48/464 ... yes 118/465 ... yes 28/467 ... yes 93/468 ... yes 39/469 ... yes 62/470 ... yes 28/471 ... yes 28/472 ... yes 48/473 ... yes 28/474 ... yes 28/475 ... yes 28/476 ... yes 93/478 ... yes 125/479 ... yes 28/481 ... yes 125/482 ... yes 125/483 ... yes 93/485 ... yes 63/486 ... yes 63/487 ... yes 63/488 ... yes 93/489 ... yes 63/490 ... yes 93/491 ... yes 39/493 ... yes 39/494 ... yes 111/495 ... yes 62/496 ... yes 62/497 ... yes 62/498 ... yes 39/499 ... yes 28/500 ... yes 123/501 ... yes 93/502 ... yes 93/503 ... yes 127/504 ... yes 91/505 ... yes 91/506 ... yes 48/507 ... yes 126/508 ... yes 126/509 ... yes 126/510 ... yes 17/511 ... yes 127/513 ... yes 101/514 ... yes 28/515 ... yes 93/516 ... yes 46/518 ... yes 15/519 ... yes 93/521 ... yes 18/522 ... yes 129/523 ... yes 125/524 ... yes 51/525 ... yes 51/526 ... yes 18/527 ... yes 15/528 ... yes 54/529 ... yes 28/530 ... yes 28/531 ... yes 93/532 ... yes 92/533 ... yes 46/534 ... yes 28/535 ... yes 39/536 ... yes 39/537 ... yes 28/538 ... yes 39/539 ... yes 28/540 ... yes 46/541 ... yes 28/542 ... yes 28/543 ... yes 28/544 ... yes 28/545 ... yes 18/548 ... yes 18/549 ... yes 46/551 ... yes 136/553 ... yes 65/554 ... yes 22/555 ... yes 77/557 ... yes 28/558 ... yes 54/560 ... yes 54/561 ... yes 28/562 ... yes 93/564 ... yes 143/565 ... yes 24/566 ... yes 144/567 ... yes 136/568 ... yes 142/569 ... yes 138/570 ... yes 138/572 ... yes 48/573 ... yes 142/574 ... yes 143/575 ... yes 144/576 ... yes 93/577 ... yes 128/580 ... yes 28/582 ... yes 93/584 ... yes 39/585 ... yes 39/587 ... yes 93/588 ... yes 93/589 ... yes 91/590 ... yes 93/591 ... yes 63/593 ... yes 93/594 ... yes 91/595 ... yes 39/596 ... yes 93/597 ... yes 156/599 ... yes 46/600 ... yes 156/601 ... yes 156/602 ... yes 39/603 ... yes 92/604 ... yes 93/605 ... yes 93/606 ... yes 157/607 ... yes 63/608 ... yes 156/609 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.2) Git version >= 2.29.0 ? ... yes (2.29.0) Git user has default SSH configuration? ... no Try fixing it: mkdir ~/gitlab-check-backup-1611332947 sudo mv /var/opt/gitlab/.ssh/id_rsa ~/gitlab-check-backup-1611332947 sudo mv /var/opt/gitlab/.ssh/id_rsa.pub ~/gitlab-check-backup-1611332947 For more information see: doc/ssh/README.md in section "SSH on the GitLab server" Please fix the error above and rerun the checks. Active users: ... 38 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... yes (7.10.2)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished