Go virtual repository

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Overview

This epic tracks the implementation of a Go virtual registry, replacing GitLab's current experimental Go proxy with a production-ready virtual registry that follows the same architecture established by the Maven, npm, and Docker virtual registries. The Go virtual registry will act as a single endpoint for resolving Go modules across multiple sources — both internal GitLab projects and remote upstreams (e.g., proxy.golang.org, private module proxies) — with proxying, caching, and unified configuration.

Why replace the Go proxy?

The current Go proxy has been stuck in experimental status due to performance concerns and architectural limitations. Meanwhile, the Go ecosystem has matured significantly — notably with the introduction of GOAUTH, which provides native authentication support for go client HTTP requests. Rather than investing further in the experimental Go proxy, a virtual registry approach gives us:

  • A consistent architecture across all supported package formats
  • Proxying and caching of Go modules from remote upstreams
  • Support for locally-hosted modules (GitLab projects as upstreams)
  • Better performance through caching — eliminating the need to clone entire repositories on each request
  • A path to GA-quality Go module support within GitLab

Background

  • The existing Go proxy is experimental and has known performance issues related to Git repository cloning on every module request
  • GitLab documentation previously (incorrectly) listed the Go proxy as a prerequisite for using projects as Go packages
  • GOAUTH investigation ([link to investigation issue]) explored authentication improvements and confirmed the feasibility of a virtual registry approach
  • Go module protocol is well-defined ($GOPROXY endpoints: $base/$module/@v/list, $base/$module/@v/$version.info, .mod, .zip) making it a strong fit for the virtual registry proxy/cache pattern

Acceptance Criteria

Complete Product Feature Set

🔄 Planned

  • [ ] Go module protocol support
    • Implement GOPROXY-compatible endpoints (/@v/list, /@v/$version.info, /@v/$version.mod, /@v/$version.zip)
    • Handle module path resolution including major version suffixes
    • Support $base/$module/@latest endpoint
    • Checksum database (sumdb) compatibility — ensure cached modules pass go client verification
  • [ ] API to configure a Go virtual registry and upstream sources
    • CRUD operations for virtual registries and upstreams
    • Support for remote upstreams (e.g., proxy.golang.org, Artifactory Go repositories, Athens, private proxies)
    • Support for locally-hosted upstreams (GitLab projects/groups exposing Go modules)
    • Priority-ordered upstream resolution
  • [ ] API to CRUD the cache
    • Cache Go module files (.info, .mod, .zip) from remote upstreams
    • Cache invalidation and manual clearing
  • [ ] Multiple registries support
    • Allow multiple Go virtual registries within a top-level group
  • [ ] UI to configure registries and upstreams
    • Leverage shared virtual registry UI components from Maven/npm/Docker implementations
    • Go-specific upstream configuration (e.g., sumdb settings, GONOSUMDB patterns)
  • [ ] UI for searching and managing the cache
    • Browse cached modules by path and version
    • View cache metadata (size, pull frequency, last accessed)
  • [ ] Ability to test the connection of a given upstream
    • Validate upstream connectivity and authentication before saving
  • [ ] Shareable upstreams functionality
    • Allow multiple Go virtual registries to reference common upstream sources
    • Shared cache storage to reduce duplication
  • [ ] Support for locally-hosted Go modules
    • Expose GitLab projects as Go module upstreams without requiring the experimental Go proxy
    • Module path hints from project configuration to optimize upstream resolution
    • Handle private project authentication natively
  • [ ] Lifecycle policies implementation
    • Delete cache records not pulled in last X days
    • Audit logs and notifications for policy-driven actions
  • [ ] Allow/Deny filtering
    • Pattern-based filtering on module paths (e.g., allow only github.com/myorg/*)
    • Version-specific filtering
    • Audit logs for blocked module requests
    • Align with GONOSUMDB / GONOSUMCHECK / GOPRIVATE conventions where appropriate
  • [ ] Root-group-level view of upstreams and their connection status
  • [ ] Statistics and Analytics
    • Cache records: pull frequency, last pull timestamp
    • Upstream usage metrics
    • Counter caches for upstream count at registry level
  • [ ] Guard against duplicated upstreams
    • Prevent same URL + same credentials within top-level group
  • [ ] Geo support
  • [ ] Complete documentation suite
    • Configuration guides for Go virtual registry setup
    • Migration guide from Go proxy to virtual registry
    • GOAUTH configuration examples for private module access
    • GOPROXY environment variable configuration for pointing go clients at the virtual registry
    • Examples for common workflows (private modules, vendoring, CI/CD integration)
  • [ ] Deprecation path for experimental Go proxy
    • Document migration steps from Go proxy → virtual registry
    • Provide timeline for Go proxy deprecation

Go-Specific Technical Considerations

  • Module path resolution: Go modules use import paths (e.g., gitlab.com/myorg/myproject) that map to Git repositories. The virtual registry needs to resolve these paths efficiently without cloning entire repositories — a key performance improvement over the current Go proxy.
  • Checksum database compatibility: The go client verifies module checksums against sum.golang.org by default. Cached modules must maintain integrity so that checksum verification passes. Consider how GONOSUMCHECK and GONOSUMDB interact with virtual registry configuration.
  • GOAUTH integration: Leverage GOAUTH for authenticating go client requests to the virtual registry, particularly for private modules. Document recommended GOAUTH configurations for GitLab.
  • Major version paths: Go's major version convention (/v2, /v3, etc.) affects module path resolution and should be handled correctly in upstream routing.
  • go-import meta tags: For locally-hosted modules, ensure proper go-import meta tag responses so that go get can discover modules served through the virtual registry.

Data & Observability Requirements

  • [ ] Full instrumentation deployed and validated
  • [ ] Usage metrics, performance data, and health indicators viewable in Tableau
  • [ ] Dashboards configured for ongoing monitoring
  • [ ] Error rate monitoring on Go virtual registry endpoints
  • [ ] Performance benchmarks for module resolution and download times (with comparison to current Go proxy)

Market Validation Requirements

  • [ ] Customer feedback collected from beta users
  • [ ] Validated with at least 3 enterprise customers using Go in production
  • [ ] Competitive analysis against JFrog Go repository and Athens proxy

Implementation Notes

Shared Infrastructure

Much of the virtual registry infrastructure is shared across formats. The Go implementation should leverage:

  • Existing virtual registry API framework (CRUD registries, upstreams, cache)
  • Shared UI components for registry management
  • Common upstream resolution and caching patterns
  • Lifecycle policy engine
  • Allow/deny filtering framework

Go-Specific Implementation

The primary new work is:

  1. Go module protocol endpoint handlers (GOPROXY-compatible API)
  2. Module path → upstream resolution logic (including major version handling)
  3. .zip, .mod, .info file caching and serving
  4. Sumdb/checksum compatibility layer
  5. GOAUTH-aware authentication
  6. Local upstream support that replaces the Git-clone approach of the current Go proxy

/cc @trizzi @10io

Edited by 🤖 GitLab Bot 🤖