Proxy dependency proxy manifest content-type (!53667) · Merge requests · GitLab.org / GitLab

Steve Abrams requested to merge 290944-dependency-proxy-content-type into master Feb 08, 2021

🔭 Context

The Dependency Proxy allows users to pull images from DockerHub through GitLab. GitLab then stores these images at the group level and will use the cached image the next time a user pulls an image.

In !52805 (merged), we added the ability for the Content-Type of an image manifest file to be stored in the database so it could be used when serving the file. Then... 💥 the Content-Type was not being returned on GitLab.com, causing the Dependency Proxy to break for all of GitLab.com, and forcing a revert in !53506 (merged).

See the original MR description if you would like a deeper description of the changes that were originally made. The revert MR reverted all changes with the exception of the original database migration that added a content_type column to dependency_proxy_manifests. It also added a migration that removed any bad data that was created during that time.

What was the problem

The problem was that on GitLab.com, we use GCP storage for object storage. GCP does not allow the setting of headers when files are downloaded, meaning, the content-type we were setting in the Rails controller never made it to the final response. The docker client would then see the incorrect content-type and politely decline to continue pulling the requested image. GCP uses whatever content-type it has assigned to the file, which will default to application/octet-stream, a value that the Docker client does not like.

🔍 What does this MR do?

The good news is, when storage is configured using other storage configurations (local file storage, S3 object storage, Minio, etc...), the Content-Type header was correctly making it through (this is why we never saw the problem since GDK generally uses minio for testing object storage). There are two ways to cause GCP to use the custom Content-Type that we have stored in the database:

When the file is stored in GCP, we can store it with the custom Content-Type value that we originally received from DockerHub.
If the Content-Type in GCP is blank, GCP will allow the response-content-type header to tell it what the correct Content-Type is. This only works if the value in GCP is already blank.

To help ease the review, I've split this MR into two commits, the first commit contains all of the changes from the original MR that were reverted. This code is identical to the code that was originally reviewed.

The second commit, !53667 (9263faa3), contains the new updates that set the file content-type in the uploader.

Screenshots (strongly suggested)

In order to have full confidence in this change, I tested against local file storage, and object storage using Minio, GCP Storage, AWS S3, and Azure blob storage. I believe GCP is the only provider that had the content-type issues, but looking at the results, all cloud providers now store the content-type, which is a good improvement.

Failing pull before the update (GCP):

docker pull gdk.test:3001/depprox/dependency_proxy/containers/nginx:latest
Error response from daemon: missing signature key

Successful pull after the update (GCP):

docker pull gdk.test:3001/depprox/dependency_proxy/containers/nginx:latest
latest: Pulling from depprox/dependency_proxy/containers/nginx
45b42c59be33: Pull complete
d0d9e9ea897e: Pull complete
66e650438339: Pull complete
76a3dfe4406b: Pull complete
410ff9d97480: Pull complete
Digest: sha256:1a53eb723d17523512bd25c27299046cfa034cce309f4ed330c943a304513f59
Status: Downloaded newer image for gdk.test:3001/depprox/dependency_proxy/containers/nginx:latest
gdk.test:3001/depprox/dependency_proxy/containers/nginx:latest

Curl request showing the incorrect header being used before the change (GCP):

TOKEN=$(curl -u 'root:' "http://gdk.test:3001/jwt/auth?account=root&scope=repository:depprox/dependency_proxy/containers/alpine:pull&service=dependency_proxy" | jq --raw-output .token) && curl -i --header "Accept: application/vnd.docker.distribution.manifest.v2+json" --header "Authorization: Bearer $TOKEN" "http://gdk.test:3001/v2/depprox/dependency_proxy/containers/alpine/manifests/latest" 2>&1 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 239 100 239 0 0 114 0 0:00:02 0:00:02 --:--:-- 114 HTTP/1.1 200 OK Accept-Ranges: bytes Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43" Cache-Control: max-age=0, private, must-revalidate, no-store Content-Length: 528 Content-Type: application/octet-stream Docker-Content-Digest: sha256:3747d4eb5e7f0825d54c8e80452f1e245e24bd715972c919d189a62da97af2ae Docker-Distribution-Api-Version: registry/2.0 Etag: "6362fb1e7488a85873909f7a756dcede" Last-Modified: Fri, 12 Feb 2021 21:21:45 GMT Pragma: no-cache Referrer-Policy: strict-origin-when-cross-origin Server: UploadServer Set-Cookie: perf_bar_enabled=true; path=/ Set-Cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IkltRmlORGRoTXpSbExUUXdOemt0TkRjMk5DMDRaR0UyTFRJM1kyWXdabUV5WTJNM01pST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--5e74c8cdbb335207028aab93bd9df387ef0fef43; path=/; expires=Tue, 12 Feb 2041 21:34:51 GMT; HttpOnly Set-Cookie: _gitlab_session_3fdd58e4697dfbfe2fe94a1e473c25da8afb14cbaf0b93d9078f18a9ce13a58b=12bedda1c618fbe1290298f1bb7ca295; path=/; HttpOnly X-Accel-Buffering: no X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: DENY X-Gitlab-Feature-Category: dependency_proxy X-Goog-Custom-Time: 1970-01-01T00:00:00Z X-Goog-Generation: 1613164905243436 X-Goog-Hash: crc32c=CA/uwQ== X-Goog-Hash: md5=Y2L7HnSIqFhzkJ96dW3O3g== X-Goog-Meta-: X-Goog-Metageneration: 2 X-Goog-Storage-Class: STANDARD X-Goog-Stored-Content-Encoding: identity X-Goog-Stored-Content-Length: 528 X-Guploader-Uploadid: ABg5-UxlpWP7aszIf2Sl1EQ93OYhqmze0fDstM_nHxaLlN8uq2L_sY_mbILUnmWpRww77e3uFgy8g6z6FXZV81rTMHc X-Permitted-Cross-Domain-Policies: none X-Request-Id: 01EYC2P0BFGKXGSWD03QBTQC3G X-Runtime: 2.414366 X-Ua-Compatible: IE=edge X-Xss-Protection: 1; mode=block Date: Fri, 12 Feb 2021 21:34:53 GMT

{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "size": 1471, "digest": "sha256:e50c909a8df2b7c8b92a6e8730e210ebe98e5082871e66edd8ef4d90838cbd25" }, "layers": [ { "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", "size": 2811321, "digest": "sha256:4c0d98bf9879488e0407f897d9dd4bf758555a78e39675e72b5124ccf12c2580" } ] }

Curl request showing the correct headers being used on a freshly pulled manifest (GCP)

TOKEN=$(curl -u 'root:' "http://gdk.test:3001/jwt/auth?account=root&scope=repository:depprox/dependency_proxy/containers/alpine:pull&service=dependency_proxy" | jq --raw-output .token) && curl -i --header "Accept: application/vnd.docker.distribution.manifest.v2+json" --header "Authorization: Bearer $TOKEN" "http://gdk.test:3001/v2/depprox/dependency_proxy/containers/alpine/manifests/latest" 2>&1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   239  100   239    0     0     22      0  0:00:10  0:00:10 --:--:--    56
HTTP/1.1 200 OK
Accept-Ranges: bytes
Alt-Svc: h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: max-age=0, private, must-revalidate, no-store
Content-Length: 528
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Docker-Content-Digest: sha256:3747d4eb5e7f0825d54c8e80452f1e245e24bd715972c919d189a62da97af2ae
Docker-Distribution-Api-Version: registry/2.0
Etag: "6362fb1e7488a85873909f7a756dcede"
Last-Modified: Fri, 12 Feb 2021 21:21:45 GMT
Pragma: no-cache
Referrer-Policy: strict-origin-when-cross-origin
Server: UploadServer
Set-Cookie: perf_bar_enabled=true; path=/
Set-Cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IklqUTNORGxpTURaaUxXTXdaR0V0TkdWaE55MWlaR1psTFdZMU9EVmlaVGszTVRZek15ST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--4ad6bf7456b437302211e6610ae6fe491546be29; path=/; expires=Tue, 12 Feb 2041 21:40:33 GMT; HttpOnly
Set-Cookie: _gitlab_session_3fdd58e4697dfbfe2fe94a1e473c25da8afb14cbaf0b93d9078f18a9ce13a58b=e95e1c01cd3f5e4ef5fa9643932d8aff; path=/; HttpOnly
X-Accel-Buffering: no
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: DENY
X-Gitlab-Feature-Category: dependency_proxy
X-Goog-Custom-Time: 1970-01-01T00:00:00Z
X-Goog-Generation: 1613164905243436
X-Goog-Hash: crc32c=CA/uwQ==
X-Goog-Hash: md5=Y2L7HnSIqFhzkJ96dW3O3g==
X-Goog-Meta-:
X-Goog-Metageneration: 2
X-Goog-Storage-Class: STANDARD
X-Goog-Stored-Content-Encoding: identity
X-Goog-Stored-Content-Length: 528
X-Guploader-Uploadid: ABg5-UwCVWf1Ab-4IubnpCNmvuc3Ur-96eyDXlXvO6bRTboYhkzDH_Bfdsv_bOyxDES11FhdFczgqJfaYwot62amLPeRFs9FCw
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 01EYC304V2P81WHN88FD71Y8B4
X-Runtime: 10.753305
X-Ua-Compatible: IE=edge
X-Xss-Protection: 1; mode=block
Date: Fri, 12 Feb 2021 21:40:34 GMT

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1471,
      "digest": "sha256:e50c909a8df2b7c8b92a6e8730e210ebe98e5082871e66edd8ef4d90838cbd25"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2811321,
         "digest": "sha256:4c0d98bf9879488e0407f897d9dd4bf758555a78e39675e72b5124ccf12c2580"
      }
   ]
}

The content-type being improperly set in GCP (before the change):

The content-type being properly set in GCP (after the change):

The content-type being properly set in AWS S3 (after the change):

The content-type being properly set in Azure Blob Storage (after the change):

Successful S3 curl after change

TOKEN=$(curl -u 'root:' "http://gdk.test:3001/jwt/auth?account=root&scope=repository:depprox/dependency_proxy/containers/alpine:pull&service=dependency_proxy" | jq --raw-output .token) && curl -i --header "Accept: application/vnd.docker.distribution.manifest.v2+json" --header "Authorization: Bearer $TOKEN" "http://gdk.test:3001/v2/depprox/dependency_proxy/containers/alpine/manifests/latest" 2>&1 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 239 100 239 0 0 111 0 0:00:02 0:00:02 --:--:-- 111 HTTP/1.1 200 OK Accept-Ranges: bytes Cache-Control: max-age=0, private, must-revalidate, no-store Content-Length: 528 Content-Type: application/vnd.docker.distribution.manifest.v2+json Docker-Content-Digest: sha256:3747d4eb5e7f0825d54c8e80452f1e245e24bd715972c919d189a62da97af2ae Docker-Distribution-Api-Version: registry/2.0 Etag: "6362fb1e7488a85873909f7a756dcede" Last-Modified: Wed, 17 Feb 2021 19:52:25 GMT Pragma: no-cache Referrer-Policy: strict-origin-when-cross-origin Server: AmazonS3 Set-Cookie: perf_bar_enabled=true; path=/ Set-Cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IklqSTJPR0UyTkRVekxUVTBNall0TkdVMk5TMWlZMk00TFdFd01HRTBNMlUxTnpOak9DST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--b269e531209f91765c474c8d8f1af52fa9fee8bf; path=/; expires=Sun, 17 Feb 2041 19:52:21 GMT; HttpOnly Set-Cookie: _gitlab_session_3fdd58e4697dfbfe2fe94a1e473c25da8afb14cbaf0b93d9078f18a9ce13a58b=ad760db74deea726ededbeac2826b699; path=/; HttpOnly X-Accel-Buffering: no X-Amz-Id-2: Dzpg4XFCZg25Gf84YaSfMSyY9JRWOwdB9gthAR5CC82/snIAQ1XjXwAEysAEQPlb67rPzvv3jX4= X-Amz-Request-Id: 88B00BFAD8EE5807 X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: DENY X-Gitlab-Feature-Category: dependency_proxy X-Permitted-Cross-Domain-Policies: none X-Request-Id: 01EYRRSXKWSKM8GTVJ447N15MT X-Runtime: 2.701905 X-Ua-Compatible: IE=edge X-Xss-Protection: 1; mode=block Date: Wed, 17 Feb 2021 19:52:24 GMT

Successful Azure curl after the change

TOKEN=$(curl -u 'root:3XH371e1zEuuUjmy7bvW' "http://gdk.test:3001/jwt/auth?account=root&scope=repository:depprox/dependency_proxy/containers/alpine:pull&service=dependency_proxy" | jq --raw-output .token) && curl -i --header "Accept: application/vnd.docker.distribution.manifest.v2+json" --header "Authorization: Bearer $TOKEN" "http://gdk.test:3001/v2/depprox/dependency_proxy/containers/alpine/manifests/latest" 2>&1 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 239 100 239 0 0 106 0 0:00:02 0:00:02 --:--:-- 106 HTTP/1.1 200 OK Accept-Ranges: bytes Cache-Control: max-age=0, private, must-revalidate, no-store Content-Length: 528 Content-Md5: Y2L7HnSIqFhzkJ96dW3O3g== Content-Type: application/vnd.docker.distribution.manifest.v2+json Docker-Content-Digest: sha256:3747d4eb5e7f0825d54c8e80452f1e245e24bd715972c919d189a62da97af2ae Docker-Distribution-Api-Version: registry/2.0 Etag: "0x8D8D380F14E8C8A" Last-Modified: Wed, 17 Feb 2021 20:16:43 GMT Pragma: no-cache Referrer-Policy: strict-origin-when-cross-origin Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 Set-Cookie: perf_bar_enabled=true; path=/ Set-Cookie: experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IklqTmxNamd6TVRNMExUSmhOMkV0TkRsaU5TMDRZMkkzTFRGaE5EZG1ZbUptTlRWa09TST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--287b9b5a5658bd1f37075659a5b66b6f36fcb411; path=/; expires=Sun, 17 Feb 2041 20:16:40 GMT; HttpOnly Set-Cookie: _gitlab_session_3fdd58e4697dfbfe2fe94a1e473c25da8afb14cbaf0b93d9078f18a9ce13a58b=88a58e377a29b2b5e2fdbb2f429bd9fd; path=/; HttpOnly X-Accel-Buffering: no X-Content-Type-Options: nosniff X-Download-Options: noopen X-Frame-Options: DENY X-Gitlab-Feature-Category: dependency_proxy X-Ms-Blob-Type: BlockBlob X-Ms-Creation-Time: Wed, 17 Feb 2021 20:16:43 GMT X-Ms-Lease-State: available X-Ms-Lease-Status: unlocked X-Ms-Request-Id: 339127cd-001e-005b-3a69-05f56e000000 X-Ms-Server-Encrypted: true X-Ms-Version: 2018-11-09 X-Permitted-Cross-Domain-Policies: none X-Request-Id: 01EYRT6E2T7EJ02FMYPRNKVDN8 X-Runtime: 2.505979 X-Ua-Compatible: IE=edge X-Xss-Protection: 1; mode=block Date: Wed, 17 Feb 2021 20:16:42 GMT

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
[-] Tested in all supported browsers
[-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

[-] Label as security and @ mention @gitlab-com/gl-security/appsec
[-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
[-] Security reports checked/validated by a reviewer from the AppSec team

Related to #290944 (closed)

Edited Feb 19, 2021 by Steve Abrams

Proxy dependency proxy manifest content-type