Go Module Proxy MVC

Background

Most dependency management systems are artifact-based, whereas Go's dependency management is source-based. That is, for most languages, releasing a new (version of a) package involves uploading some form of artifact to a package registry/repository. Some languages upload source and some upload binaries. However, Go has no notion of a package registry or repository. Instead of a package registry, Go fetches a package directly from its version control system.

Go packages necessarily cannot have name collisions because the name of a Go package must include the URL from which it can be fetched (e.g. golang.org/x/text). Only the standard library is allowed to omit a domain (e.g. net/http).

Go 1.13 introduced the notion of a Go proxy. Prior to this, or with the proxy disabled, go fetches packages directly from their VCS. The Go proxy protocol is a simple HTTP protocol that facilitates caching, avoids interfacing directly with a VCS, and allows go to fetch only what is needed instead of cloning the entire source repository.

What does this MR do?

This MR implements a Go proxy MVC, which is the equivalent of a Go package manager. The MVC will not act as a dependency proxy and will only serve modules that correspond to projects on the instance that have enabled (Go) package management. The MVC will not serve legacy Go packages, i.e. those which do not have a go.mod file.

This MR will implement an API endpoint conformant to the Go module proxy protocol. This endpoint will initially be available at /api/v4/projects/:id/packages/go.

Closes #27376 (closed)

Concerns

• Handling pseudo-versions and tags that might mask those
  • Per discussion and go's approach, tags that match the pseudo-version pattern will be ignored
• Omitting module versions with an invalid go.mod
  • Per discussion, err on the side of omission, because relaxing restrictions is backwards compatible but the inverse is not
• What response should be returned when the request does not match a visible project?
  • 401 when unauthenticated and 404 when authenticated, to support tooling that only prompts for credentials on 401.

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

Beyond the MVC

Buyer: Engineering (full support for Go modules stored on GitLab)

• Proxy modules from projects requiring authentication Done
• Group-level configuration and API endpoint #213900
• Instance-level configuration and API endpoint #213902
• Proxy legacy packages (those that do not define any modules, i.e. having no go.mod files) #213907
• Module discoverability under the Packages view #213770
• Format go.mod with hyperlinks to dependencies #212708 (closed)

Buyer: Director/Executive (complete control over Go development ecosystem)

• Dependency proxy for 3rd party packages (those external to the instance) #213907
• Checksum database (sum.golang.org) #213907
• Index/package feed (index.golang.org) #213907