The default Faraday URL params encoder breaks integration with Google Cloud CDN for the Container Registry
Problem
This is the root cause for gitlab-com/gl-infra/production#6381 (closed).
We have discovered that the Faraday lib used by the ContainerRegistry::Client
class (used to interact with the Container Registry) is escaping query parameters in URLs by default.
This is fine for most situations, but the Google Cloud CDN signed URLs (to where the container registry redirects Rails for obtaining the configuration payload of container images) must not be URL encoded.
While debugging gitlab-com/gl-infra/production#6381 (closed), we noticed that URLs ending with =
were being sent to Google as %3D
. This causes Google to reject the request because the last portion of the URL is the signature, so it can't be touched.
Example Cloud CDN URL:
https://cdn.registry.gitlab-static.net/gitlab/docker/registry/v2/blobs/sha256/2d/2df365faf0e3007f983fadd7a65ba51d41b488eb2ed8fc70f4bf97043cfea560/data?Expires=1645202555&KeyName=gprd-registry-cdn&Signature=66dyddgqweg12b43431wd=
Right now, Faraday will pick this URL and replace =
with %3D
before invoking it:
https://cdn.registry.gitlab-static.net/gitlab/docker/registry/v2/blobs/sha256/2d/2df365faf0e3007f983fadd7a65ba51d41b488eb2ed8fc70f4bf97043cfea560/data?Expires=1645202555&KeyName=gprd-registry-cdn&Signature=66dyddgqweg12b43431wd%3D
Notes
- This issue only occurs in GitLab.com, where the registry Cloud CDN feature is fully rolled out and enabled by default;
- This issue only occurs when the originating request (either through the UI or API) comes from outside GCP. For internal requests created within the application (such as for cleanup policies) this is not a problem because they are redirected to GCS and not Cloud CDN;
- This is only a problem when obtaining the
config
of aContainerRegistry::Tag
. This is commonly used to determine the creation timestamp of an image.
Solution
The easiest fix is to instruct Faraday not to URL escape query parameters, at least for the redirection URLs that happen during the ContainerRegistry::Client.blob
method.