Dependency proxy fails to pull OCI image indexes
Summary
Trying to pull images packaged as an OCI Image Index through the dependency proxy returns a 404 error.
This has been reported by several customers with multiple different images, such as:
grafana/mimirmoby/buildkitbrowserless/chrometimbru31/node-chrome
What all these images have in common is that they are multi-architecture images, packaged in an OCI index. The dependency proxy does not support this image type, thus the failure.
When attempting to pull the Grafana/Mimir images through dependency proxy results in a 404 error.
Steps to reproduce
- Authenticate with your group's dependency proxy
- Pull one of the known affected images through the dependency proxy. You should see the following error:
$ docker pull gitlab.com/gitlab-gold/dependency_proxy/containers/grafana/mimir Using default tag: latest Error response from daemon: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value - Pull the same image via
dockerand observe there are no issues:$ docker pull grafana/mimir Using default tag: latest latest: Pulling from grafana/mimir Digest: sha256:63d891f40d575825d4d22a595a5e71c2bb7b4786bb8539e3d893e0fe0f3139bf Status: Downloaded newer image for grafana/mimir:latest docker.io/grafana/mimir:latest
What is the current bug behavior?
The image returns a 400 level error when pulled via dependency proxy and does not reflect in the list of cached images.
What is the expected correct behavior?
The image is pulled successfully and is reflected as cached in the dependency proxy.
Relevant logs and/or screenshots
Click to expand
jlopes@jlopes-workpc ~ $ docker login gitlab.com
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/jlopes/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
jlopes@jlopes-workpc ~ $ docker pull gitlab.com/gitlab-gold/dependency_proxy/containers/grafana/mimir
Using default tag: latest
Error response from daemon: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE html>\n<html>\n<head>\n <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n <title>The page you're looking for could not be found (404)</title>\n <style>\n body {\n color: #666;\n text-align: center;\n font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n margin: auto;\n font-size: 14px;\n }\n\n h1 {\n font-size: 56px;\n line-height: 100px;\n font-weight: 400;\n color: #456;\n }\n\n h2 {\n font-size: 24px;\n color: #666;\n line-height: 1.5em;\n }\n\n h3 {\n color: #456;\n font-size: 20px;\n font-weight: 400;\n line-height: 28px;\n }\n\n hr {\n max-width: 800px;\n margin: 18px auto;\n border: 0;\n border-top: 1px solid #EEE;\n border-bottom: 1px solid white;\n }\n\n img {\n max-width: 40vw;\n display: block;\n margin: 40px auto;\n }\n\n a {\n line-height: 100px;\n font-weight: 400;\n color: #4A8BEE;\n font-size: 18px;\n text-decoration: none;\n }\n\n .container {\n margin: auto 20px;\n }\n\n .go-back {\n display: none;\n }\n\n </style>\n</head>\n\n<body>\n <a href=\"/\">\n <img src='data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjAwIiBoZWlnaHQ9IjE5MiIgdmlld0JveD0iMCAwIDI1IDI0IiBmaWxsPSJub25lIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogIDxwYXRoIGQ9Im0yNC41MDcgOS41LS4wMzQtLjA5TDIxLjA4Mi41NjJhLjg5Ni44OTYgMCAwIDAtMS42OTQuMDkxbC0yLjI5IDcuMDFINy44MjVMNS41MzUuNjUzYS44OTguODk4IDAgMCAwLTEuNjk0LS4wOUwuNDUxIDkuNDExLjQxNiA5LjVhNi4yOTcgNi4yOTcgMCAwIDAgMi4wOSA3LjI3OGwuMDEyLjAxLjAzLjAyMiA1LjE2IDMuODY3IDIuNTYgMS45MzUgMS41NTQgMS4xNzZhMS4wNTEgMS4wNTEgMCAwIDAgMS4yNjggMGwxLjU1NS0xLjE3NiAyLjU2LTEuOTM1IDUuMTk3LTMuODkuMDE0LS4wMUE2LjI5NyA2LjI5NyAwIDAgMCAyNC41MDcgOS41WiIKICAgICAgICBmaWxsPSIjRTI0MzI5Ii8+CiAgPHBhdGggZD0ibTI0LjUwNyA5LjUtLjAzNC0uMDlhMTEuNDQgMTEuNDQgMCAwIDAtNC41NiAyLjA1MWwtNy40NDcgNS42MzIgNC43NDIgMy41ODQgNS4xOTctMy44OS4wMTQtLjAxQTYuMjk3IDYuMjk3IDAgMCAwIDI0LjUwNyA5LjVaIgogICAgICAgIGZpbGw9IiNGQzZEMjYiLz4KICA8cGF0aCBkPSJtNy43MDcgMjAuNjc3IDIuNTYgMS45MzUgMS41NTUgMS4xNzZhMS4wNTEgMS4wNTEgMCAwIDAgMS4yNjggMGwxLjU1NS0xLjE3NiAyLjU2LTEuOTM1LTQuNzQzLTMuNTg0LTQuNzU1IDMuNTg0WiIKICAgICAgICBmaWxsPSIjRkNBMzI2Ii8+CiAgPHBhdGggZD0iTTUuMDEgMTEuNDYxYTExLjQzIDExLjQzIDAgMCAwLTQuNTYtMi4wNUwuNDE2IDkuNWE2LjI5NyA2LjI5NyAwIDAgMCAyLjA5IDcuMjc4bC4wMTIuMDEuMDMuMDIyIDUuMTYgMy44NjcgNC43NDUtMy41ODQtNy40NDQtNS42MzJaIgogICAgICAgIGZpbGw9IiNGQzZEMjYiLz4KPC9zdmc+Cg==' alt=\"GitLab\"/>\n </a>\n <h1>\n 404\n </h1>\n <div class=\"container\">\n <h3>The page could not be found or you don't have permission to view it.</h3>\n <hr />\n <p>The resource that you are attempting to access does not exist or you don't have the necessary permissions to view it.</p>\n <p>Make sure the address is correct and that the page hasn't moved.</p>\n <p>Please contact your GitLab administrator if you think this is a mistake.</p>\n <a href=\"javascript:history.back()\" class=\"js-go-back go-back\">Go back</a>\n </div>\n <script>\n (function () {\n var goBack = document.querySelector('.js-go-back');\n\n if (history.length > 1) {\n goBack.style.display = 'inline';\n }\n })();\n </script>\n</body>\n</html>\n"
jlopes@jlopes-workpc ~ $ docker pull grafana/mimir
Using default tag: latest
latest: Pulling from grafana/mimir
Digest: sha256:91f0c6a484fc5946f400bd736aefa218067e0b386e5398f95d17a283b22bce09
Status: Downloaded newer image for grafana/mimir:latest
docker.io/grafana/mimir:latest
Output of checks
This bug happens on GitLab.com.
Solution
Expand the manifest media types supported by the Dependency Proxy to include the one that identifies OCI Image Indexes: application/vnd.oci.image.index.v1+json.
The current accepted media types advertised by the Dependency Proxy are injected here. Instead of expanding the ContainerRegistry::Client::ACCEPTED_TYPES, we should instead define a custom list of accepted types for the Dependency Proxy. Why? The registry base client is used for most or all of the registry functionality spread across the codebase, such as for the GitLab UI/API and Geo replication. Therefore, extending the list of supported media types there will have an impact beyond the dependency proxy. There are already issues to tackle the support for multi-arch images in other areas of the application (e.g. #232815 (closed)), and those should be tackled separately for satefy (and iteration).
For this reason, we should define a e.g. ACCEPTED_MANIFEST_TYPES list, within the DependencyProxy module. This list should include the following media types:
ContainerRegistry::BaseClient::DOCKER_DISTRIBUTION_MANIFEST_V2_TYPEContainerRegistry::BaseClient::OCI_MANIFEST_V1_TYPEContainerRegistry::BaseClient::OCI_DISTRIBUTION_INDEX_TYPE
We should then use this new list to populate the Accept header on the DependencyProxy requests sent to DockerHub.
Doing so will fix this issue and both image indexes and the underlying platform-specific images will show up on the list of dependency proxied images in the UI:
Workaround
While the fix does not land, the possible workaround is to identify the image digest of the platform-specific image that you're looking to pull and then use that instead of the tag/digest of the multi-arch image.
For example, looking at the grafana/mimir image. On DockerHub we can see the following:
In this case, to pull the latest version of this image you'd do:
docker pull mygitlab.com/mygroup/dependency_proxy/containers/grafana/mimir:latestDoing so would result in an error, because this is a multi-architecture image based on OCI indexes. To workaround this, we need to use the digest of the variant that we want to pull. Imagining it's linux/amd64, we'd use the digest highlighted in green. To see the non-truncated version you can click on it and see the full version on the next page:
Then, you can grab that digest and do:
docker pull mygitlab.com/mygroup/dependency_proxy/containers/grafana/mimir@sha256:9de73a35e12cee57c8d5863ec5de2c462646c439dffa1f22bd243e3deb8e9a2aThis should work as now you're not trying to pull a multi-arch image (OCI index) but rather a single platform-specific image variant.