Draft: Forward RubyGems package requests to rubygems.org via a package setting

Hannah Bakerrequested to merge
601752-rubygem-forwarding-no-proxy into master

⚠️ Superseded — proof of concept, not for merge. This MR established and secured approval for the RubyGems forwarding mechanism (design discussion: concern → alignment → approval). The agreed approach now ships as a series of smaller, reviewable MRs — start here: !244621 (merged) — tracked under #601752. This MR will not be merged; it stays open so the approval discussion remains accessible.

Slice MR Sub-issue
Setting (data + API) !244621 (merged) #605488
Forwarding + admin UI !244623 (closed) #605489
Group + package-list UI !244626 (closed) #605490

What this MR does

When a project's RubyGems registry does not have a requested gem, this MR forwards the request to rubygems.org, so bundle install, gem fetch, and gem install can resolve and download upstream gems through GitLab. Forwarding is turned on by a new application/group/project setting (rubygems_package_requests_forwarding), the same way npm, PyPI, and Maven forwarding already work.

This MR is only the forwarding mechanism — it does not include any Dependency Firewall (policy checks / blocking). See "Out of scope" below.

Background (no prior knowledge needed)

A RubyGems client (Bundler or the gem CLI) is pointed at a "source" — here, a GitLab project's registry. When it needs a gem it sends a sequence of HTTP requests to that source: first to resolve what versions and dependencies exist, then to download the .gem file (the "tarball").

Today, if the gem isn't stored in the GitLab project, GitLab returns 404 and the client gives up on GitLab. Forwarding changes that "not found" into a redirect to rubygems.org, so the client keeps working. This is the same "act as a mirror" behaviour npm/PyPI/Maven already have.

How forwarding works

When the setting is on and a gem isn't stored locally, GitLab responds with a 302 redirect to the matching rubygems.org URL, for each request in the client's flow:

Endpoint Used by Not-found behaviour with forwarding on
GET .../versions, .../info/:gem_name (compact index) Bundler 302 → rubygems.org
GET .../:file_name (spec index: specs.4.8.gz …) gem 302 → rubygems.org
GET .../quick/Marshal.4.8/:file_name (gemspec) gem 302 → rubygems.org
GET .../gems/:file_name (tarball) both 302 → rubygems.org

GitLab redirects (302); it does not download and re-serve gem data, so there is no added egress through GitLab.

Gating: an application setting, not the firewall

Forwarding is controlled by a new setting, rubygems_package_requests_forwarding, at the instance level with per-group/project override and a lock_ sibling — identical in shape to npm_package_requests_forwarding / pypi_package_requests_forwarding. When the setting is off (and on CE), behaviour is unchanged: not-found gems return 404, nothing is forwarded.

Default: OFF (deliberate divergence from npm/PyPI/Maven)

This setting defaults to false. npm, PyPI, and Maven all default their equivalent forwarding setting to true, so on those an upgrade silently starts redirecting not-found requests to the public registry. We intentionally do not do that for RubyGems: defaulting on would change behaviour for every existing instance on upgrade — a not-found gem that returns 404 today would suddenly start resolving from rubygems.org — with unknown impact on current customer usage. Shipping default-off makes this MR a no-op for existing instances until an admin explicitly opts in, and lets the rollout be a conscious decision rather than an upgrade side effect.

Note the tradeoff: unlike Maven, there is no separate feature flag on the forwarding path, so the setting itself is the only switch. If we later want staged enablement (default the setting on but keep actual forwarding dark behind a flag, as Maven does with maven_central_request_forwarding), that would be a follow-up. For now, default-off is the safety mechanism.

gem install and the retired RubyGems dependency API

This is the one non-obvious part, so here it is in full.

gem install begins by asking the source "do you support the dependency API?" with a request to /api/v1/dependencies. That endpoint is an old bulk-resolution API that rubygems.org has retired — it now returns 404 with the body "The dependency API has gone away." (deprecation: https://blog.rubygems.org/2023/02/22/dependency-api-deprecation.html, briefly delayed https://blog.rubygems.org/2023/04/07/dependency-api-deprecation-delayed.html, later completed).

Because rubygems.org answers that probe with 404, the gem client falls back to the older "Marshal index" (specs.4.8.gz → gemspec → tarball) and installs successfully. So gem install works fine without the dependency API — those Marshal-index endpoints are exactly the ones this MR forwards.

The problem is that GitLab still implements /api/v1/dependencies (it serves local gems' deps), so it answers that probe with 200. The gem client takes that to mean "this API works here," commits to it, then gets a 404 for the upstream gem (which isn't stored locally) and treats it as "that gem does not exist" — so it aborts without falling back, and never reaches the forwarded endpoints.

The fix: when forwarding is on, GitLab returns 404 on /api/v1/dependencies — i.e. it behaves like rubygems.org. The gem client then falls back to the Marshal index (which this MR forwards) and gem install of an upstream gem works. When forwarding is off, the endpoint is unchanged and gem install of a gem stored in the project keeps resolving locally as before.

(References: gem install command — https://guides.rubygems.org/command-reference/#gem-install)

Behaviour summary

Client, for an upstream gem forwarding off / CE forwarding on
bundle install 404 (fails) resolves + installs
gem fetch 404 (fails) downloads
gem install 404 (fails) resolves + installs (via the fallback above)
any of the above for a gem in the registry served locally served locally

Out of scope (follow-up MR)

No Dependency Firewall code is in this MR. The firewall — evaluating a forwarded gem against policy and blocking a disallowed one with 403 — is a separate follow-up that plugs a check into the forwarding path, gated on the dependency_firewall licence. This MR is the Package Registry plumbing that follow-up builds on.

Commit structure

  1. Forwarding + the rubygems_package_requests_forwarding setting — Bundler and gem fetch work here; gem install of an upstream gem still aborts.
  2. /api/v1/dependencies returns 404 when forwarding is on — gem install now falls back to the forwarded Marshal index and works. Isolated so the before/after is demonstrable.

How to validate locally

Enable both flags/settings on a project, then (token needs read_package_registry):

# Bundler:
ruby -e 'require "bundler/inline"; gemfile do
  source "http://__token__:<token>@gdk.test:3000/api/v4/projects/<id>/packages/rubygems"
  gem "colorize"
end'
# gem CLI:
gem install colorize --clear-sources --source "http://__token__:<token>@gdk.test:3000/api/v4/projects/<id>/packages/rubygems"

Turn the setting off and re-run: every request returns 404, nothing is forwarded.

References

Edited by Hannah Baker

Merge request reports