Serve RubyGems spec index files through the API

Tim Rizzirequested to merge
feature/rubygems-spec-files-download into master

Summary

Implements the RubyGems spec index download endpoint for the package registry — the final piece of #299267 (closed).

This serves the spec index files generated by the previously merged Packages::Rubygems::CreateSpecFilesService / CreateSpecFilesWorker so RubyGems clients can resolve and install gems from a project's registry:

GET /api/v4/projects/:id/packages/rubygems/:file_name

where :file_name is one of specs.4.8.gz, latest_specs.4.8.gz, or prerelease_specs.4.8.gz.

The feature remains gated behind the per-project :rubygem_packages feature flag (experimental).

Implementation Details

The endpoint:

  1. Authorizes the request with authorize_read_package!(project).
  2. Looks up the stored Packages::Rubygems::SpecFile by (project_id, file_name).
  3. Serves it via present_carrierwave_file! (direct download / Workhorse send-url, consistent with the existing gemspec and gem download routes).

Cache-miss self-heal: if no spec file exists for the project, the endpoint enqueues CreateSpecFilesWorker and returns 404. This regenerates the index for projects whose packages predate spec file generation — for example, gems pushed while the feature flag was disabled — so a subsequent request succeeds. The worker is idempotent! and deduplicated, so a burst of gem install retries collapses into a single regeneration.

To satisfy the CodeReuse/Worker cop (workers cannot be enqueued directly from an API endpoint), the enqueue lives in a new API::Helpers::Packages::Rubygems module, mirroring the npm metadata-cache helper (lib/api/helpers/packages/npm.rb).

Database

No migrations or schema changes. This is a read-only path.

The lookup uses the Rails dynamic finder SpecFile.find_by_project_id_and_file_name(project_id, file_name), which resolves against the existing unique index index_packages_rubygems_spec_files_on_project_id_and_file_name. No new scope is introduced and no write is performed, so this should not require a dedicated database review.

SELECT "packages_rubygems_spec_files".*
FROM "packages_rubygems_spec_files"
WHERE "packages_rubygems_spec_files"."project_id" = 278964
  AND "packages_rubygems_spec_files"."file_name" = 'specs.4.8.gz'
LIMIT 1;

This is the same single-row index lookup whose plan was documented and reviewed in the spec-file generation MR (Index Scan using index_packages_rubygems_spec_files_on_project_id_and_file_name, ~0.03 ms execution on a DLE clone).

Affected code paths

  • lib/api/rubygem_packages.rb — implements the previously stubbed GET :file_name route; adds route_setting :authorization, permissions: :read_ruby_gem, boundary_type: :project to match the sibling read routes.
  • lib/api/helpers/packages/rubygems.rb (new) — enqueue_create_spec_files_worker(project) helper.
  • doc/api/packages/rubygems.md — "Download a spec index file" section.

Regenerated artifacts (standard side effects of adding route_setting :authorization + a documented route):

  • config/authz/routes/authorization_todo.txt — route removed (now has granular authorization).
  • doc/auth/tokens/fine_grained_access_tokens_rest.md — route added as Read / Project.
  • doc/api/openapi/openapi_v2.yaml, doc/api/openapi/openapi_v3.yaml — regenerated endpoint description.

Test Plan

bundle exec rspec spec/requests/api/rubygem_packages_spec.rb

bundle exec rubocop \
  lib/api/rubygem_packages.rb \
  lib/api/helpers/packages/rubygems.rb \
  spec/requests/api/rubygem_packages_spec.rb

Request specs cover:

  • The visibility / role / token authorization matrix (personal access token, job token, deploy token).
  • Successful download — asserts the served file (application/octet-stream, X-Sendfile).
  • Cache miss — asserts CreateSpecFilesWorker.perform_async is enqueued and the response is 404.
  • Feature flag disabled and package feature disabled → 404.
  • Granular token permission (read_ruby_gem).

Functional Testing

  1. Enable the :rubygem_packages feature flag for a test project and push one or more .gem files (including a prerelease such as 1.0.0.pre).

  2. After upload processing completes (the worker runs via the existing hooks), download each index:

    curl --header "PRIVATE-TOKEN: <token>" \
  --url "http://gdk.test:3000/api/v4/projects/<id>/packages/rubygems/specs.4.8.gz" \
  --output specs.4.8.gz
ruby -e 'require "zlib"; pp Marshal.load(Zlib.gunzip(File.read("specs.4.8.gz")))'

  3. Verify the three files contain the expected entries:

    • specs.4.8.gz — released versions only.
    • latest_specs.4.8.gz — latest released version per gem.
    • prerelease_specs.4.8.gz — prerelease versions only.

  4. Cache-miss self-heal: request a spec file for a flag-enabled project that has packages but no generated spec file → expect 404 and confirm CreateSpecFilesWorker is enqueued; retry after it runs → expect the file.

  5. End-to-end: gem install <gem> --source <project registry> now resolves against the served index.

Merge request reports