Commit f9bc8d20 authored by Stan Hu's avatar Stan Hu

Merge branch '27376-go-package-mvc' into 'master'

Go Module Proxy MVC

Closes #27376

See merge request !27746
parents a57593b7 d5e63a6d
Pipeline #149063484 canceled with stages
......@@ -66,6 +66,12 @@ class Settings < Settingslogic
(base_url(gitlab) + [gitlab.relative_url_root]).join('')
end
def build_gitlab_go_url
# "Go package paths are not URLs, and do not include port numbers"
# https://github.com/golang/go/issues/38213#issuecomment-607851460
"#{gitlab.host}#{gitlab.relative_url_root}"
end
def kerberos_protocol
kerberos.https ? "https" : "http"
end
......
......@@ -13,6 +13,7 @@ The Packages feature allows GitLab to act as a repository for the following:
| [Conan Repository](../../user/packages/conan_repository/index.md) | The GitLab Conan Repository enables every project in GitLab to have its own space to store [Conan](https://conan.io/) packages. | 12.4+ |
| [Maven Repository](../../user/packages/maven_repository/index.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ |
| [NPM Registry](../../user/packages/npm_registry/index.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ |
| [Go Proxy](../../user/packages/go_proxy/index.md) | The Go proxy for GitLab enables every project in GitLab to be fetched with the [Go proxy protocol](https://proxy.golang.org/). | 13.1+ |
Don't you see your package management system supported yet?
Please consider contributing
......
# GitLab Go Proxy **(PREMIUM)**
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/27376) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.1.
> - It's deployed behind a feature flag, disabled by default.
> - It's disabled on GitLab.com.
> - It's not recommended for production use.
> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-the-go-proxy). **(PREMIUM)**
With the Go proxy for GitLab, every project in GitLab can be fetched with the
[Go proxy protocol](https://proxy.golang.org/).
## Prerequisites
### Enable the Go proxy
The Go proxy for GitLab is under development and not ready for production use, due to
[potential performance issues with large repositories](https://gitlab.com/gitlab-org/gitlab/-/issues/218083).
It is deployed behind a feature flag that is **disabled by default**.
[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md)
can enable it for your instance.
To enable it:
```ruby
Feature.enable(:go_proxy) # or
```
To disable it:
```ruby
Feature.disable(:go_proxy)
```
To enable or disable it for specific projects:
```ruby
Feature.enable(:go_proxy, Project.find(1))
Feature.disable(:go_proxy, Project.find(2))
```
### Enable the Package Registry
The Package Registry is enabled for new projects by default. If you cannot find
the **{package}** **Packages > List** entry under your project's sidebar, verify
the following:
1. Your GitLab administrator has
[enabled support for the Package Registry](../../../administration/packages/index.md). **(PREMIUM ONLY)**
1. The Package Registry is [enabled for your project](../index.md).
NOTE: **Note:**
GitLab does not currently display Go modules in the **Packages Registry** of a project.
Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213770) for details.
### Fetch modules from private projects
`go` does not support transmitting credentials over insecure connections. The
steps below work only if GitLab is configured for HTTPS.
1. Configure Go to include HTTP basic authentication credentials when fetching
from the Go proxy for GitLab.
1. Configure Go to skip downloading of checksums for private GitLab projects
from the public checksum database.
#### Enable Request Authentication
Create a [personal access token](../../profile/personal_access_tokens.md) with
the `api` or `read_api` scope and add it to
[`~/.netrc`](https://ec.haxx.se/usingcurl/usingcurl-netrc):
```netrc
machine <url> login <username> password <token>
```
`<url>` should be the URL of the GitLab instance, for example `gitlab.com`.
`<username>` and `<token>` should be your username and the personal access
token, respectively.
#### Disable checksum database queries
Go can be configured to query a checksum database for module checksums. Go 1.13
and later query `sum.golang.org` by default. This fails for modules that are not
public and thus not accessible to `sum.golang.org`. To resolve this issue, set
`GONOSUMDB` to a comma-separated list of projects or namespaces for which Go
should not query the checksum database. For example, `go env -w
GONOSUMDB=gitlab.com/my/project` persistently configures Go to skip checksum
queries for the project `gitlab.com/my/project`.
Checksum database queries can be disabled for arbitrary prefixes or disabled
entirely. However, checksum database queries are a security mechanism and as
such they should be disabled selectively and only when necessary. `GOSUMDB=off`
or `GONOSUMDB=*` disables checksum queries entirely. `GONOSUMDB=gitlab.com`
disables checksum queries for all projects hosted on GitLab.com.
## Add GitLab as a Go proxy
NOTE: **Note:**
To use a Go proxy, you must be using Go 1.13 or later.
The available proxy endpoints are:
- Project - can fetch modules defined by a project - `/api/v4/projects/:id/packages/go`
Go's use of proxies is configured with the `GOPROXY` environment variable, as a
comma separated list of URLs. Go 1.14 adds support for comma separated list of
URLs. Go 1.14 adds support for using `go env -w` to manage Go's environment
variables. For example, `go env -w GOPROXY=...` writes to `$GOPATH/env`
(which defaults to `~/.go/env`). `GOPROXY` can also be configured as a normal
environment variable, with RC files or `export GOPROXY=...`.
The default value of `$GOPROXY` is `https://proxy.golang.org,direct`, which
tells `go` to first query `proxy.golang.org` and fallback to direct VCS
operations (`git clone`, `svc checkout`, etc). Replacing
`https://proxy.golang.org` with a GitLab endpoint will direct all fetches
through GitLab. Currently GitLab's Go proxy does not support dependency
proxying, so all external dependencies will be handled directly. If GitLab's
endpoint is inserted before `https://proxy.golang.org`, then all fetches will
first go through GitLab. This can help avoid making requests for private
packages to the public proxy, but `GOPRIVATE` is a much safer way of achieving
that.
For example, with the following configuration, Go will attempt to fetch modules
from 1) GitLab project 1234's Go module proxy, 2) `proxy.golang.org`, and
finally 3) directly with Git (or another VCS, depending on where the module
source is hosted).
```shell
go env -w GOPROXY=https://gitlab.com/api/v4/projects/1234/packages/go,https://proxy.golang.org,direct
```
## Release a module
Go modules and module versions are handled entirely with Git (or SVN, Mercurial,
and so on). A module is a repository containing Go source and a `go.mod` file. A
version of a module is a Git tag (or equivalent) that is a valid [semantic
version](https://semver.org), prefixed with 'v'. For example, `v1.0.0` and
`v1.3.2-alpha` are valid module versions, but `v1` or `v1.2` are not.
Go requires that major versions after v1 involve a change in the import path of
the module. For example, version 2 of the module `gitlab.com/my/project` must be
imported and released as `gitlab.com/my/project/v2`.
For a complete understanding of Go modules and versioning, see [this series of
blog posts](https://blog.golang.org/using-go-modules) on the official Go
website.
## Valid modules and versions
The GitLab Go proxy will ignore modules and module versions that have an invalid
`module` directive in their `go.mod`. Go requires that a package imported as
`gitlab.com/my/project` can be accessed with that same URL, and that the first
line of `go.mod` is `module gitlab.com/my/project`. If `go.mod` names a
different module, compilation will fail. Additionally, Go requires, for major
versions after 1, that the name of the module have an appropriate suffix, for
example `gitlab.com/my/project/v2`. If the `module` directive does not also have
this suffix, compilation will fail.
Go supports 'pseudo-versions' that encode the timestamp and SHA of a commit.
Tags that match the pseudo-version pattern are ignored, as otherwise they could
interfere with fetching specific commits using a pseudo-version. Pseudo-versions
follow one of three formats:
- `vX.0.0-yyyymmddhhmmss-abcdefabcdef`, when no earlier tagged commit exists for X.
- `vX.Y.Z-pre.0.yyyymmddhhmmss-abcdefabcdef`, when most recent prior tag is vX.Y.Z-pre.
- `vX.Y.(Z+1)-0.yyyymmddhhmmss-abcdefabcdef`, when most recent prior tag is vX.Y.Z.
......@@ -21,6 +21,7 @@ The Packages feature allows GitLab to act as a repository for the following:
| [NPM Registry](npm_registry/index.md) **(PREMIUM)** | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ |
| [NuGet Repository](nuget_repository/index.md) **(PREMIUM)** | The GitLab NuGet Repository will enable every project in GitLab to have its own space to store [NuGet](https://www.nuget.org/) packages. | 12.8+ |
| [PyPi Repository](pypi_repository/index.md) **(PREMIUM)** | The GitLab PyPi Repository will enable every project in GitLab to have its own space to store [PyPi](https://pypi.org/) packages. | 12.10+ |
| [Go Proxy](go_proxy/index.md) **(PREMIUM)** | The Go proxy for GitLab enables every project in GitLab to be fetched with the [Go proxy protocol](https://proxy.golang.org/). | 13.1+ |
## Enable the Package Registry for your project
......@@ -116,7 +117,6 @@ are adding support for [PHP](https://gitlab.com/gitlab-org/gitlab/-/merge_reques
| [Conda](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | Secure and private local Conda repositories. |
| [CRAN](https://gitlab.com/gitlab-org/gitlab/-/issues/36892) | Deploy and resolve CRAN packages for the R language. |
| [Debian](https://gitlab.com/gitlab-org/gitlab/-/issues/5835) | Host and provision Debian packages. |
| [Go](https://gitlab.com/gitlab-org/gitlab/-/issues/9773) | Resolve Go dependencies from and publish your Go packages to GitLab. |
| [Opkg](https://gitlab.com/gitlab-org/gitlab/-/issues/36894) | Optimize your work with OpenWrt using Opkg repositories. |
| [P2](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | Host all your Eclipse plugins in your own GitLab P2 repository. |
| [Puppet](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | Configuration management meets repository management with Puppet repositories. |
......
# frozen_string_literal: true
module Packages
module Go
class ModuleFinder
include ::API::Helpers::Packages::Go::ModuleHelpers
attr_reader :project, :module_name
def initialize(project, module_name)
module_name = Pathname.new(module_name).cleanpath.to_s
@project = project
@module_name = module_name
end
def execute
return if @module_name.blank? || !@module_name.start_with?(gitlab_go_url)
module_path = @module_name[gitlab_go_url.length..].split('/')
project_path = project.full_path.split('/')
module_project_path = module_path.shift(project_path.length)
return unless module_project_path == project_path
Packages::GoModule.new(@project, @module_name, module_path.join('/'))
end
private
def gitlab_go_url
@gitlab_go_url ||= Settings.build_gitlab_go_url + '/'
end
end
end
end
# frozen_string_literal: true
module Packages
module Go
class VersionFinder
include ::API::Helpers::Packages::Go::ModuleHelpers
attr_reader :mod
def initialize(mod)
@mod = mod
end
def execute
@mod.project.repository.tags
.filter { |tag| semver? tag }
.map { |tag| @mod.version_by(ref: tag) }
.filter { |ver| ver.valid? }
end
def find(target)
case target
when String
if pseudo_version? target
semver = parse_semver(target)
commit = pseudo_version_commit(@mod.project, semver)
Packages::GoModuleVersion.new(@mod, :pseudo, commit, name: target, semver: semver)
else
@mod.version_by(ref: target)
end
when Gitlab::Git::Ref
@mod.version_by(ref: target)
when ::Commit, Gitlab::Git::Commit
@mod.version_by(commit: target)
else
raise ArgumentError.new 'not a valid target'
end
end
end
end
end
# frozen_string_literal: true
class Packages::GoModule
include Gitlab::Utils::StrongMemoize
attr_reader :project, :name, :path
def initialize(project, name, path)
@project = project
@name = name
@path = path
end
def versions
strong_memoize(:versions) { Packages::Go::VersionFinder.new(self).execute }
end
def version_by(ref: nil, commit: nil)
raise ArgumentError.new 'no filter specified' unless ref || commit
raise ArgumentError.new 'ref and commit are mutually exclusive' if ref && commit
if commit
return version_by_sha(commit) if commit.is_a? String
return version_by_commit(commit)
end
return version_by_name(ref) if ref.is_a? String
version_by_ref(ref)
end
def path_valid?(major)
m = /\/v(\d+)$/i.match(@name)
case major
when 0, 1
m.nil?
else
!m.nil? && m[1].to_i == major
end
end
def gomod_valid?(gomod)
if Feature.enabled?(:go_proxy_disable_gomod_validation, @project)
return gomod&.start_with?("module ")
end
gomod&.split("\n", 2)&.first == "module #{@name}"
end
private
def version_by_name(name)
# avoid a Gitaly call if possible
if strong_memoized?(:versions)
v = versions.find { |v| v.name == ref }
return v if v
end
ref = @project.repository.find_tag(name) || @project.repository.find_branch(name)
return unless ref
version_by_ref(ref)
end
def version_by_ref(ref)
# reuse existing versions
if strong_memoized?(:versions)
v = versions.find { |v| v.ref == ref }
return v if v
end
commit = ref.dereferenced_target
semver = Packages::SemVer.parse(ref.name, prefixed: true)
Packages::GoModuleVersion.new(self, :ref, commit, ref: ref, semver: semver)
end
def version_by_sha(sha)
commit = @project.commit_by(oid: sha)
return unless ref
version_by_commit(commit)
end
def version_by_commit(commit)
Packages::GoModuleVersion.new(self, :commit, commit)
end
end
# frozen_string_literal: true
class Packages::GoModuleVersion
include Gitlab::Utils::StrongMemoize
include ::API::Helpers::Packages::Go::ModuleHelpers
VALID_TYPES = %i[ref commit pseudo].freeze
attr_reader :mod, :type, :ref, :commit
delegate :major, to: :@semver, allow_nil: true
delegate :minor, to: :@semver, allow_nil: true
delegate :patch, to: :@semver, allow_nil: true
delegate :prerelease, to: :@semver, allow_nil: true
delegate :build, to: :@semver, allow_nil: true
def initialize(mod, type, commit, name: nil, semver: nil, ref: nil)
raise ArgumentError.new("invalid type '#{type}'") unless VALID_TYPES.include? type
raise ArgumentError.new("mod is required") unless mod
raise ArgumentError.new("commit is required") unless commit
if type == :ref
raise ArgumentError.new("ref is required") unless ref
elsif type == :pseudo
raise ArgumentError.new("name is required") unless name
raise ArgumentError.new("semver is required") unless semver
end
@mod = mod
@type = type
@commit = commit
@name = name if name
@semver = semver if semver
@ref = ref if ref
end
def name
@name || @ref&.name
end
def full_name
"#{mod.name}@#{name || commit.sha}"
end
def gomod
strong_memoize(:gomod) do
if strong_memoized?(:blobs)
blob_at(@mod.path + '/go.mod')
elsif @mod.path.empty?
@mod.project.repository.blob_at(@commit.sha, 'go.mod')&.data
else
@mod.project.repository.blob_at(@commit.sha, @mod.path + '/go.mod')&.data
end
end
end
def archive
suffix_len = @mod.path == '' ? 0 : @mod.path.length + 1
Zip::OutputStream.write_buffer do |zip|
files.each do |file|
zip.put_next_entry "#{full_name}/#{file[suffix_len...]}"
zip.write blob_at(file)
end
end
end
def files
strong_memoize(:files) do
ls_tree.filter { |e| !excluded.any? { |n| e.start_with? n } }
end
end
def excluded
strong_memoize(:excluded) do
ls_tree
.filter { |f| f.end_with?('/go.mod') && f != @mod.path + '/go.mod' }
.map { |f| f[0..-7] }
end
end
def valid?
@mod.path_valid?(major) && @mod.gomod_valid?(gomod)
end
private
def blob_at(path)
return if path.nil? || path.empty?
path = path[1..] if path.start_with? '/'
blobs.find { |x| x.path == path }&.data
end
def blobs
strong_memoize(:blobs) { @mod.project.repository.batch_blobs(files.map { |x| [@commit.sha, x] }) }
end
def ls_tree
strong_memoize(:ls_tree) do
path =
if @mod.path.empty?
'.'
else
@mod.path
end
@mod.project.repository.gitaly_repository_client.search_files_by_name(@commit.sha, path)
end
end
end
# frozen_string_literal: true
class Packages::SemVer
attr_accessor :major, :minor, :patch, :prerelease, :build
def initialize(major = 0, minor = 0, patch = 0, prerelease = nil, build = nil, prefixed: false)
@major = major
@minor = minor
@patch = patch
@prerelease = prerelease
@build = build
@prefixed = prefixed
end
def prefixed?
@prefixed
end
def ==(other)
self.class == other.class &&
self.major == other.major &&
self.minor == other.minor &&
self.patch == other.patch &&
self.prerelease == other.prerelease &&
self.build == other.build
end
def to_s
s = "#{prefixed? ? 'v' : ''}#{major || 0}.#{minor || 0}.#{patch || 0}"
s += "-#{prerelease}" if prerelease
s += "+#{build}" if build
s
end
def self.match(str, prefixed: false)
return unless str&.start_with?('v') == prefixed
str = str[1..] if prefixed
Gitlab::Regex.semver_regex.match(str)
end
def self.match?(str, prefixed: false)
!match(str, prefixed: prefixed).nil?
end
def self.parse(str, prefixed: false)
m = match str, prefixed: prefixed
return unless m
new(m[1].to_i, m[2].to_i, m[3].to_i, m[4], m[5], prefixed: prefixed)
end
end
# frozen_string_literal: true
module Packages
module Go
class ModuleVersionPresenter
def initialize(version)
@version = version
end
def name
@version.name
end
def time
@version.commit.committed_date
end
end
end
end
---
title: Implement Go module proxy MVC (package manager for Go)
merge_request: 27746
author: Ethan Reesor
type: added
# frozen_string_literal: true
module API
class GoProxy < Grape::API
helpers ::API::Helpers::PackagesHelpers
helpers ::API::Helpers::Packages::Go::ModuleHelpers
# basic semver, except case encoded (A => !a)
MODULE_VERSION_REGEX = /v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(?:-([-.!a-z0-9]+))?(?:\+([-.!a-z0-9]+))?/.freeze
MODULE_VERSION_REQUIREMENTS = { module_version: MODULE_VERSION_REGEX }.freeze
before { require_packages_enabled! }
helpers do
def find_project!(id)
# based on API::Helpers::Packages::BasicAuthHelpers#authorized_project_find!
project = find_project(id)
return project if project && can?(current_user, :read_project, project)
if current_user
not_found!('Project')
else
unauthorized!
end
end
def find_module
not_found! unless Feature.enabled?(:go_proxy, user_project)
module_name = case_decode params[:module_name]
bad_request!('Module Name') if module_name.blank?
mod = ::Packages::Go::ModuleFinder.new(user_project, module_name).execute
not_found! unless mod
mod
end
def find_version
module_version = case_decode params[:module_version]
ver = ::Packages::Go::VersionFinder.new(find_module).find(module_version)
not_found! unless ver&.valid?
ver
rescue ArgumentError
not_found!
end
end
params do
requires :id, type: String, desc: 'The ID of a project'
requires :module_name, type: String, desc: 'Module name', coerce_with: ->(val) { CGI.unescape(val) }
end
route_setting :authentication, job_token_allowed: true, basic_auth_personal_access_token: true
resource :projects, requirements: API::NAMESPACE_OR_PROJECT_REQUIREMENTS do
before do
authorize_read_package!
authorize_packages_feature!
end
namespace ':id/packages/go/*module_name/@v' do
desc 'Get all tagged versions for a given Go module' do
detail 'See `go help goproxy`, GET $GOPROXY/<module>/@v/list. This feature was introduced in GitLab 13.1.'
end
get 'list' do
mod = find_module
content_type 'text/plain'
mod.versions.map { |t| t.name }.join("\n")
end
desc 'Get information about the given module version' do
detail 'See `go help goproxy`, GET $GOPROXY/<module>/@v/<version>.info. This feature was introduced in GitLab 13.1.'
success EE::API::Entities::GoModuleVersion
end
params do
requires :module_version, type: String, desc: 'Module version'
end
get ':module_version.info', requirements: MODULE_VERSION_REQUIREMENTS do
ver = find_version
present ::Packages::Go::ModuleVersionPresenter.new(ver), with: EE::API::Entities::GoModuleVersion
end
desc 'Get the module file of the given module version' do
detail 'See `go help goproxy`, GET $GOPROXY/<module>/@v/<version>.mod. This feature was introduced in GitLab 13.1.'
end
params do
requires :module_version, type: String, desc: 'Module version'
end
get ':module_version.mod', requirements: MODULE_VERSION_REQUIREMENTS do
ver = find_version
content_type 'text/plain'
ver.gomod
end
desc 'Get a zip of the source of the given module version' do
detail 'See `go help goproxy`, GET $GOPROXY/<module>/@v/<version>.zip. This feature was introduced in GitLab 13.1.'
end
params do
requires :module_version, type: String, desc: 'Module version'
end
get ':module_version.zip', requirements: MODULE_VERSION_REQUIREMENTS do
ver = find_version
content_type 'application/zip'
env['api.format'] = :binary
header['Content-Disposition'] = ActionDispatch::Http::ContentDisposition.format(disposition: 'attachment', filename: ver.name + '.zip')
header['Content-Transfer-Encoding'] = 'binary'
status :ok
body ver.archive.string
end
end
end
end