Skip to content

Add git.SendGoArchive

This adds a new Rails-to-Workhorse 'method' specifically for handling repository archive downloads from the Go proxy. As noted in !27746 and #218083, archive generation should be done in the Workhorse and not in rails. However, the Go proxy must be able to exclude paths from the archive, and git.SendArchive does not support this.

From Dependency Management in Go > 'Module' vs 'Package':

  • Any subdirectories of a module that themselves are modules are distinct, separate modules and are excluded from the containing module.
    • Given a module repo, if repo/sub contains a go.mod file then repo/sub and any files contained therein are a separate module and not a part of repo.

From go help goproxy:

GET $GOPROXY/<module>/@v/<version>.zip returns the zip archive for that version of the given module.

For a given path in the repository that is a module, any subdirectories of that path that are themselves modules are excluded from the 'parent' module and must be excluded from GitLab's response. For example, given that dir/go.mod and dir/sub/go.mod exist, a request for <domain>/<project full path>/mod@<version>.zip must exclude dir/sub/*.

Updates gitlab#218083

Implementation

Go requires a ZIP but reading a ZIP requires random access, so SendGoArchive asks Gitaly for a TAR GZ and converts it to a ZIP:

  1. Call Gitaly, requesting a TAR GZ of the given path in the repository
  2. Open a pipe, attach a ZIP writer to the write side and copy the read side to the HTTP response
  3. Uncompress the GZIP stream and for each TAR entry:
    1. Skip if the entry name matches an excluded path
    2. Create a ZIP entry
    3. Copy the TAR entry to the ZIP entry

I refactored git.SendArchive, moving some of the logic into a new function, so that it can be shared with git.SendGoArchive.

Edited by Ethan Reesor

Merge request reports