Draft: Fully support Go modules in the Packages UI and API

Background

Go uses a source-based dependency management system, whereas most other dependency management systems are artifact-based. This is to say, Go dependencies are ultimately fetched directly from their source VCS repository, but dependencies in other systems are artifacts that have been uploaded to a package repository. Another unique feature of the Go ecosystem is the name of a package (excluding stdlib) must be a valid URL, sans the scheme (e.g. golang.org/x/text). Thus, Go modules are defined by the source repository and have unique names.

For the artifact-based dependency management systems that GitLab supports (i.e. all of them except Go), database entries are created when the user pushes a package. However, Go dependency management is VCS-based, which in the context of GitLab means Git-based, thus a Git tag is a module release and there is no other way to release modules (technically anything is possible, but that would be a Bad Idea).

What does this MR do?

This MR adds an API endpoint, PUT /api/v4/projects/:id/packages/refresh. This endpoint scans the repository's tags and blobs for valid Go modules and creates package entities for any Go module (versions) that do not yet have a package.

This MR incorporates Go modules into the Packages UI and API.

Closes #213770

Concerns and/or future work

• Transition from synchronous record creation to a sidekiq job.
• Fix SQL N+1 #219308 (closed)
• Reduce Gitaly calls #218083, #219311
• When the user pushes a tag, queue a sidekiq update for that ref. #220626
• When the user navigates to a Go proxy resource, compare the generated resources to Package entries, and possibly queue a sidekiq update. #220628 (closed)
• When the user navigates to certain paths or calls certain API resources, compare the list of tags to the list of Packages, and possibly queue a sidekiq update. #220629
  • /my/project/-/packages
  • /api/v4/projects/:id/packages

Screenshots

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team