Dependency proxy fails to pull OCI image indexes
<!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> ### Summary Trying to pull images packaged as an [OCI Image Index](https://github.com/opencontainers/image-spec/blob/main/image-index.md) through the dependency proxy returns a 404 error. This has been reported by several customers with multiple different images, such as: - `grafana/mimir` - `moby/buildkit` - `browserless/chrome` - `timbru31/node-chrome` What all these images have in common is that they are multi-architecture images, packaged in an OCI index. The dependency proxy does not support this image type, thus the failure. When attempting to pull the [Grafana/Mimir](https://hub.docker.com/r/grafana/mimir) images through dependency proxy results in a 404 error. ### Steps to reproduce <!-- Describe how one can reproduce the issue - this is very important. Please use an ordered list. --> 1. Authenticate with your group's dependency proxy 2. Pull one of the known affected images through the dependency proxy. You should see the following error: ```sh $ docker pull gitlab.com/gitlab-gold/dependency_proxy/containers/grafana/mimir Using default tag: latest Error response from daemon: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value ``` 3. Pull the same image via `docker` and observe there are no issues: ```sh $ docker pull grafana/mimir Using default tag: latest latest: Pulling from grafana/mimir Digest: sha256:63d891f40d575825d4d22a595a5e71c2bb7b4786bb8539e3d893e0fe0f3139bf Status: Downloaded newer image for grafana/mimir:latest docker.io/grafana/mimir:latest ``` ### What is the current *bug* behavior? The image returns a 400 level error when pulled via dependency proxy and does not reflect in the list of cached images. ### What is the expected *correct* behavior? The image is pulled successfully and is reflected as cached in the dependency proxy. ### Relevant logs and/or screenshots <details><summary>Click to expand</summary> ``` jlopes@jlopes-workpc ~ $ docker login gitlab.com Authenticating with existing credentials... WARNING! Your password will be stored unencrypted in /home/jlopes/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded jlopes@jlopes-workpc ~ $ docker pull gitlab.com/gitlab-gold/dependency_proxy/containers/grafana/mimir Using default tag: latest Error response from daemon: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE html>\n<html>\n<head>\n <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n <title>The page you're looking for could not be found (404)</title>\n <style>\n body {\n color: #666;\n text-align: center;\n font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n margin: auto;\n font-size: 14px;\n }\n\n h1 {\n font-size: 56px;\n line-height: 100px;\n font-weight: 400;\n color: #456;\n }\n\n h2 {\n font-size: 24px;\n color: #666;\n line-height: 1.5em;\n }\n\n h3 {\n color: #456;\n font-size: 20px;\n font-weight: 400;\n line-height: 28px;\n }\n\n hr {\n max-width: 800px;\n margin: 18px auto;\n border: 0;\n border-top: 1px solid #EEE;\n border-bottom: 1px solid white;\n }\n\n img {\n max-width: 40vw;\n display: block;\n margin: 40px auto;\n }\n\n a {\n line-height: 100px;\n font-weight: 400;\n color: #4A8BEE;\n font-size: 18px;\n text-decoration: none;\n }\n\n .container {\n margin: auto 20px;\n }\n\n .go-back {\n display: none;\n }\n\n </style>\n</head>\n\n<body>\n <a href=\"/\">\n <img src='data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjAwIiBoZWlnaHQ9IjE5MiIgdmlld0JveD0iMCAwIDI1IDI0IiBmaWxsPSJub25lIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPgogIDxwYXRoIGQ9Im0yNC41MDcgOS41LS4wMzQtLjA5TDIxLjA4Mi41NjJhLjg5Ni44OTYgMCAwIDAtMS42OTQuMDkxbC0yLjI5IDcuMDFINy44MjVMNS41MzUuNjUzYS44OTguODk4IDAgMCAwLTEuNjk0LS4wOUwuNDUxIDkuNDExLjQxNiA5LjVhNi4yOTcgNi4yOTcgMCAwIDAgMi4wOSA3LjI3OGwuMDEyLjAxLjAzLjAyMiA1LjE2IDMuODY3IDIuNTYgMS45MzUgMS41NTQgMS4xNzZhMS4wNTEgMS4wNTEgMCAwIDAgMS4yNjggMGwxLjU1NS0xLjE3NiAyLjU2LTEuOTM1IDUuMTk3LTMuODkuMDE0LS4wMUE2LjI5NyA2LjI5NyAwIDAgMCAyNC41MDcgOS41WiIKICAgICAgICBmaWxsPSIjRTI0MzI5Ii8+CiAgPHBhdGggZD0ibTI0LjUwNyA5LjUtLjAzNC0uMDlhMTEuNDQgMTEuNDQgMCAwIDAtNC41NiAyLjA1MWwtNy40NDcgNS42MzIgNC43NDIgMy41ODQgNS4xOTctMy44OS4wMTQtLjAxQTYuMjk3IDYuMjk3IDAgMCAwIDI0LjUwNyA5LjVaIgogICAgICAgIGZpbGw9IiNGQzZEMjYiLz4KICA8cGF0aCBkPSJtNy43MDcgMjAuNjc3IDIuNTYgMS45MzUgMS41NTUgMS4xNzZhMS4wNTEgMS4wNTEgMCAwIDAgMS4yNjggMGwxLjU1NS0xLjE3NiAyLjU2LTEuOTM1LTQuNzQzLTMuNTg0LTQuNzU1IDMuNTg0WiIKICAgICAgICBmaWxsPSIjRkNBMzI2Ii8+CiAgPHBhdGggZD0iTTUuMDEgMTEuNDYxYTExLjQzIDExLjQzIDAgMCAwLTQuNTYtMi4wNUwuNDE2IDkuNWE2LjI5NyA2LjI5NyAwIDAgMCAyLjA5IDcuMjc4bC4wMTIuMDEuMDMuMDIyIDUuMTYgMy44NjcgNC43NDUtMy41ODQtNy40NDQtNS42MzJaIgogICAgICAgIGZpbGw9IiNGQzZEMjYiLz4KPC9zdmc+Cg==' alt=\"GitLab\"/>\n </a>\n <h1>\n 404\n </h1>\n <div class=\"container\">\n <h3>The page could not be found or you don't have permission to view it.</h3>\n <hr />\n <p>The resource that you are attempting to access does not exist or you don't have the necessary permissions to view it.</p>\n <p>Make sure the address is correct and that the page hasn't moved.</p>\n <p>Please contact your GitLab administrator if you think this is a mistake.</p>\n <a href=\"javascript:history.back()\" class=\"js-go-back go-back\">Go back</a>\n </div>\n <script>\n (function () {\n var goBack = document.querySelector('.js-go-back');\n\n if (history.length > 1) {\n goBack.style.display = 'inline';\n }\n })();\n </script>\n</body>\n</html>\n" jlopes@jlopes-workpc ~ $ docker pull grafana/mimir Using default tag: latest latest: Pulling from grafana/mimir Digest: sha256:91f0c6a484fc5946f400bd736aefa218067e0b386e5398f95d17a283b22bce09 Status: Downloaded newer image for grafana/mimir:latest docker.io/grafana/mimir:latest ``` </details> ### Output of checks <!-- If you are reporting a bug on GitLab.com, uncomment below --> This bug happens on GitLab.com. ### Solution Expand the manifest media types supported by the Dependency Proxy to include the one that identifies OCI Image Indexes: `application/vnd.oci.image.index.v1+json`. The current accepted media types advertised by the Dependency Proxy are injected [here](https://gitlab.com/gitlab-org/gitlab/blob/81cbdc44f65ddf488ac9e05a00609c63d8eac68a/app/controllers/groups/dependency_proxy_for_containers_controller.rb#L175-175). Instead of expanding the [`ContainerRegistry::Client::ACCEPTED_TYPES`](https://gitlab.com/gitlab-org/gitlab/blob/81cbdc44f65ddf488ac9e05a00609c63d8eac68a/lib/container_registry/base_client.rb#L15-15), we should instead define a custom list of accepted types for the Dependency Proxy. Why? The registry base client is used for most or all of the registry functionality spread across the codebase, such as for the GitLab UI/API and Geo replication. Therefore, extending the list of supported media types there will have an impact beyond the dependency proxy. There are already issues to tackle the support for multi-arch images in other areas of the application (e.g. https://gitlab.com/gitlab-org/gitlab/-/issues/232815), and those should be tackled separately for satefy (and iteration). For this reason, we should define a e.g. `ACCEPTED_MANIFEST_TYPES` list, within the [`DependencyProxy`](https://gitlab.com/gitlab-org/gitlab/blob/81cbdc44f65ddf488ac9e05a00609c63d8eac68a/app/models/dependency_proxy.rb#L2-2) module. This list should include the following media types: - [`ContainerRegistry::BaseClient::DOCKER_DISTRIBUTION_MANIFEST_V2_TYPE`](https://gitlab.com/gitlab-org/gitlab/blob/81cbdc44f65ddf488ac9e05a00609c63d8eac68a/lib/container_registry/base_client.rb#L9-9) - [`ContainerRegistry::BaseClient::OCI_MANIFEST_V1_TYPE`](https://gitlab.com/gitlab-org/gitlab/blob/81cbdc44f65ddf488ac9e05a00609c63d8eac68a/lib/container_registry/base_client.rb#L12-12) - [`ContainerRegistry::BaseClient::OCI_DISTRIBUTION_INDEX_TYPE`](https://gitlab.com/gitlab-org/gitlab/blob/81cbdc44f65ddf488ac9e05a00609c63d8eac68a/lib/container_registry/base_client.rb#L11-11) We should then use this new list to populate the `Accept` header on the DependencyProxy requests sent to DockerHub. Doing so will fix this issue and both image indexes and the underlying platform-specific images will show up on the list of dependency proxied images in the UI: ![image](/uploads/e368a6e06b0edc8320556d9c51600df5/image.png) ### Workaround While the fix does not land, the possible workaround is to identify the image digest of the platform-specific image that you're looking to pull and then use that instead of the tag/digest of the multi-arch image. For example, looking at the [`grafana/mimir`](https://hub.docker.com/r/grafana/mimir/tags) image. On DockerHub we can see the following: ![Screenshot_2023-03-13_at_16.04.59](/uploads/cf9594bb49c7b1d285a118b4db91cdee/Screenshot_2023-03-13_at_16.04.59.png) In this case, to pull the `latest` version of this image you'd do: ```sh docker pull mygitlab.com/mygroup/dependency_proxy/containers/grafana/mimir:latest ``` Doing so would result in an error, because this is a multi-architecture image based on OCI indexes. To workaround this, we need to use the digest of the variant that we want to pull. Imagining it's `linux/amd64`, we'd use the digest highlighted in green. To see the non-truncated version you can click on it and see the full version on the next page: ![Screenshot_2023-03-13_at_16.05.46](/uploads/e4fe3544761e4efd901c9d4c7951f9b3/Screenshot_2023-03-13_at_16.05.46.png) Then, you can grab that digest and do: ```sh docker pull mygitlab.com/mygroup/dependency_proxy/containers/grafana/mimir@sha256:9de73a35e12cee57c8d5863ec5de2c462646c439dffa1f22bd243e3deb8e9a2a ``` This should work as now you're not trying to pull a multi-arch image (OCI index) but rather a single platform-specific image variant.
issue