Skip to content

Maven dependency proxy: handle gracefully network errors

David Fernandez requested to merge 430193-handle-refused-connections into master

🔭 Context

We're working on the very first version of the dependency proxy for packages. See #407460 (comment 1373731852) for all the details from the technical investigation.

At the core, the concept is right simple. GitLab will act with as a proxy. Basically, users can pull packages through it and GitLab will be a pull-through cache.

Package Manager clients (npm, maven, ...) <-> GitLab <-> External Package Registry

Because, GitLab is in the middle (aka proxy) of the package transport, we can leverage the GitLab Package registry to use it as a cache. In other words, before contacting the external package registry, we can check the local project registry to check if the package is already there. If that's the case, we can return it directly.

In this area, the cache has quite some logic: if a cache entry is found, we check for its coherence. To keep things simple, the backend will HEAD the file on the external package registry to check if the file has not changed. If that's the case, we're all good and we can return the file from the GitLab package registry.

In this MR, we're going to focus hard on edge cases where the HEAD requests fails. Thus, we don't know if we have the same file or not. In those cases, we implemented a "best effort" approach: we still return the file because there are high chances that it is the correct one. This helps to increase the reliability. You can imagine using GitLab to proxy a Maven external registry. If that registry is down, users can still pull packages out of the dependency proxy as long as the package exists in the GitLab package registry.

During our verifications on staging on this part we noticed:

  • If the external registry refuses the connection, the backend will end up in an 5XX error. This is quite bad 😱
  • If the external registry doesn't reply at all, we're going to wait quite a long time before getting back the file from the GitLab package registry.

That's issue Maven dependency proxy: handle gracefully a con... (#430193 - closed).

On this MR, we're going to improve the handling of both cases.

🔬 What does this MR do and why?

  • Update the verify etag service of the dependency proxy so that when the external registry is having network hiccups, we properly handle the situation. The expected behavior is that we return the existing file from the GitLab package registry.
  • Update the related specs.

The entire Maven dependency proxy is behind a feature flag. Rollout issue: #415218 (closed). Thus, we don't have a changelog here. It's basically a bug fix from a bug that we noticed during the verification on staging.

🖼 Screenshots or screen recordings

Well, the maven dependency proxy is made to be used by maven clients which are CLIs. So, no 🌈 UI.

⚙ How to set up and validate locally

The set up is a bit involved as we need an external registry where we can manipulate things to simulate network hiccups.

For this, we're going to use an AWS lightsail server that will have a dummy ruby server that will serve a single file. We will then set up the dependency proxy in the local GitLab instance and pull the file through it.

Let's get started 💪

  1. Open the AWS ligthsail.
  2. Create an instance. Select the smallest one with Ubuntu.
  3. Once it is running, open the web terminal / shell / whatever that thing is called:
    • sudo apt update
    • sudo apt install ruby -y
    • mkdir -p srv/com/my/company/1.2.3
    • cd srv/com/my/company/1.2.3
    • echo "bananas!" > test.txt
    • cd ../../../..
    • ruby -run -e httpd . -p 8081
  4. On the instance "Networking" tab, add a rule: Allow TCP 8081.

Our dummy external registry is now ready.

Let's setup our local GitLab. The maven dependency proxy has a few requirements:

  1. have packages -> enabled set to true in gitlab.yml.
  2. have dependency_proxy -> enabled set to true in gitlab.yml.
  3. have the packages feature enabled in the project's settings. Settings -> General -> Visiblity, project features, permissions -> Package registry (checkbox enabled.)
  4. have a GitLab license. Premium or more.
  5. have the related feature flag turned on: 
    Feature.enable(:packages_dependency_proxy_maven)

All good, now, let's configure the dependency proxy and create its settings to point to our cloud server.

  1. Have a private project ready.
  2. Have a PAT ready. access level maintainer or more.
  3. Let's setup the dependency proxy in rails console: 
    Project.find(<project_id>).create_dependency_proxy_packages_setting!(enabled: true, maven_external_registry_url: 'http://<lightsail instance IP>:8081')

Last steps. The situation we focus on this MR is where the requested file exists in the Package Registry. Let's pull the file so that the dependency proxy will publish it to the GitLab package registry for us. Additionally, this will check that our setup is all good. In a local shell:

$ curl "http://<GitLab username>:<GitLab PAT>@gdk.test:8000/api/v4/projects/<project id>/dependency_proxy/packages/maven/com/my/company/1.2.3/test.txt"
bananas!

You can verify in the package registry of <project_id> that you will have a maven package with the test.txt file.

All ready, let's go through our cases.

1️⃣ External registry refuses the connection

For this, simply shutdown the external registry (the ruby webrick server).

On master:

$ curl "http://<GitLab username>:<GitLab PAT>@gdk.test:8000/api/v4/projects/<project id>/dependency_proxy/packages/maven/com/my/company/1.2.3/test.txt"
{"message":"500 Internal Server Error"}

💥

With this MR:

$ curl "http://<GitLab username>:<GitLab PAT>@gdk.test:8000/api/v4/projects/<project id>/dependency_proxy/packages/maven/com/my/company/1.2.3/test.txt"
bananas!

-> The dependency proxy received the error but did the best effort here and returned the file from the GitLab package registry.

In the log/application_json.log:

{"severity":"ERROR","time":"2023-11-03T16:13:18.833Z","correlation_id":"01HEAZBJABNFFQ5W7SNB9JJWJB","meta.caller_id":"GET /api/:version/projects/:id/dependency_proxy/packages/maven/*path/:file_name","meta.remote_ip":"XXXXX","meta.feature_category":"package_registry","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","service_class":"DependencyProxy::Packages::Maven::VerifyPackageFileEtagService","project_id":241,"message":"External registry is not available"}

2️⃣ External registry doesn't answer back

(start the webrick server again if you come from 1️⃣).

To not answer back to a connection, simply remove the TCP 8081 rule from the Networking tab on the instance.

On master:

$ curl "http://<GitLab username>:<GitLab PAT>@gdk.test:8000/api/v4/projects/<project id>/dependency_proxy/packages/maven/com/my/company/1.2.3/test.txt"
{"message":"500 Internal Server Error"}

(after ~15 seconds)

With this MR:

$ curl "http://<GitLab username>:<GitLab PAT>@gdk.test:8000/api/v4/projects/<project id>/dependency_proxy/packages/maven/com/my/company/1.2.3/test.txt"
bananas!

(after ~5 seconds)

-> We wait a shorter amount of time. ✅

In the log/application_json.log:

{"severity":"ERROR","time":"2023-11-03T16:11:50.641Z","correlation_id":"01HEAZ8QABGX68JBS6ZRS6RBYM","meta.caller_id":"GET /api/:version/projects/:id/dependency_proxy/packages/maven/*path/:file_name","meta.remote_ip":"XXXX","meta.feature_category":"package_registry","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","service_class":"DependencyProxy::Packages::Maven::VerifyPackageFileEtagService","project_id":241,"message":"External registry is not available"}

3️⃣ (Bonus) File not found

(Put back the networking rule)

Remove the file from the external registry and pull it.

On master:

$ curl "http://<GitLab username>:<GitLab PAT>@gdk.test:8000/api/v4/projects/<project id>/dependency_proxy/packages/maven/com/my/company/1.2.3/test.txt"
bananas!

-> the file is served from the GitLab package registry.

With this MR:

$ curl "http://<GitLab username>:<GitLab PAT>@gdk.test:8000/api/v4/projects/<project id>/dependency_proxy/packages/maven/com/my/company/1.2.3/test.txt"
bananas!

-> the file is served from the GitLab package registry.

We can see that this case is already supported on master but we can also see that this MR doesn't change the behavior in this situation ✅

In the log/application_json.log:

{"severity":"ERROR","time":"2023-11-03T16:15:36.607Z","correlation_id":"01HEAZFWFMKAF6D8PSA5VV37P1","meta.caller_id":"GET /api/:version/projects/:id/dependency_proxy/packages/maven/*path/:file_name","meta.remote_ip":"XXXX","meta.feature_category":"package_registry","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","service_class":"DependencyProxy::Packages::Maven::VerifyPackageFileEtagService","project_id":241,"message":"Received 404 from external registry"}

♻ Post test steps

  • Don't forget to delete the lightsail instance.

🛴 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Fernandez

Merge request reports