RubyGems spec generation
Implementation plan (2 MRs)
-
Create a service or setup to generate the various spec files. 3
-
Return the files from the endpoints. 1
Routes implemented
GET /api/v4/projects/27/packages/rubygems/:file_name
GET "/api/v4/projects/27/packages/rubygems/specs.4.8.gz"
GET "/api/v4/projects/27/packages/rubygems/latest_specs.4.8.gz"
GET "/api/v4/projects/27/packages/rubygems/prerelease_specs.4.8.gz"
Notes
The spec file is an index of all gems available in the repository (in our case the project). https://blog.packagecloud.io/eng/2015/12/15/rubygem-index-internals/ is a great overview of what the indexes are and how they can be generated. There are three total. We should be able to directly use the Gem module to build the index http://docs.seattlerb.org/rubygems/Gem/Indexer.html. We can use Gem::Indexer.build_indices
but we might need to override gem_file_list
or map_gems_to_specs
since we don’t deal with directories and will need to pull the filenames from the database and the specs themselves possible from the database.
Helpful references I found while researching the index:
- https://thoughtbot.com/blog/fetching-source-index-for-http-rubygems-org
- https://andre.arko.net/2014/03/28/the-new-rubygems-index-format/
We can use https://github.com/rubygems/rubygems.org/blob/master/app/jobs/indexer.rb as an example of how to dynamically generate the index. Looking at https://github.com/rubygems/rubygems.org/blob/36d5ba22b281c10a13d62b170cb2c1c19a1253cd/app/models/version.rb#L102, it looks like we should be able to use data that exists in the DB to generate these: package.name, package.version, package.rubygems_metadatum.platform
. Rubygems.org uses S3: https://github.com/rubygems/rubygems.org/blob/36d5ba22b281c10a13d62b170cb2c1c19a1253cd/lib/rubygem_fs.rb#L7
You can read an existing unzipped specs file using irb
:
m = Marshal.load(Gem.gunzip(File.read("/path/to/latest_specs.gz")))
m.first
=> ["_", #<Gem::Version "1.4">, "ruby"]
m[4000]
=> ["agoo", #<Gem::Version "2.14.0">, "ruby"]
m[4000][1]
=> #<Gem::Version "2.14.0">
m[4000][1].version
=> "2.14.0"
References
- Install notes which include some info on the spec downloads are here: #216517 (comment 485004481)
- A summary of all routes are listed here: #216517 (comment 485886685)
- Main RubyGems issue: #803
- Investigation issue: #216517 (closed)