Skip to content

Create Packages from Go module versions

Ethan Reesor requested to merge ethan.reesor/contrib/gitlab:220628-go-proxy-packages into master

Background

Go uses a source-based dependency management system, whereas most other dependency management systems are artifact-based. This is to say, Go dependencies are ultimately fetched directly from their source VCS repository, but dependencies in other systems are artifacts that have been uploaded to a package repository. Another unique feature of the Go ecosystem is the name of a package (excluding stdlib) must be a valid URL, sans the scheme (e.g. golang.org/x/text). Thus, Go modules are defined by the source repository and have unique names.

For the artifact-based dependency management systems that GitLab supports (i.e. all of them except Go), database entries are created when the user pushes a package. However, Go dependency management is VCS-based, which in the context of GitLab means Git-based, thus a Git tag is a module release and there is no other way to release modules (technically anything is possible, but that would be a Bad Idea).

What does this MR do?

Per this comment, this MR adds:

  • A golang package type for Packages::Package !41712 (merged)
  • An index to ensure only one Go Package exists per (project, name, version)
  • A service to create Go Packages and PackageFiles
  • A worker to create Go Packages and PackageFiles asynchronously
  • Use Packages and PackageFiles as a cache for the Go proxy #254818
  • Schedule the worker from the Go proxy on cache miss #254818

Concerns and/or future work

DB Migration

Up

== 20210106061254 AddUniqueIndexForGolangPackages: migrating ==================
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:packages_packages, [:project_id, :name, :version], {:unique=>true, :where=>"package_type = 8", :name=>"index_packages_on_project_id_name_version_unique_when_golang", :algorithm=>:concurrently})
   -> 0.0098s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- add_index(:packages_packages, [:project_id, :name, :version], {:unique=>true, :where=>"package_type = 8", :name=>"index_packages_on_project_id_name_version_unique_when_golang", :algorithm=>:concurrently})
   -> 0.0119s
-- execute("RESET ALL")
   -> 0.0004s
== 20210106061254 AddUniqueIndexForGolangPackages: migrated (0.0231s) =========

Down

== 20210106061254 AddUniqueIndexForGolangPackages: reverting ==================
-- transaction_open?()
   -> 0.0000s
-- indexes(:packages_packages)
   -> 0.0106s
-- execute("SET statement_timeout TO 0")
   -> 0.0002s
-- remove_index(:packages_packages, {:algorithm=>:concurrently, :name=>"index_packages_on_project_id_name_version_unique_when_golang"})
   -> 0.0033s
-- execute("RESET ALL")
   -> 0.0003s
== 20210106061254 AddUniqueIndexForGolangPackages: reverted (0.0152s) =========

Queries review

!34558 (comment 530533329)

Exist

https://console.postgres.ai/shared/904e8286-6c40-451b-b4a2-574a55fa78de

Query:

SELECT 1 AS one
  FROM "packages_packages"
 WHERE "packages_packages"."project_id" = 25155009
   AND "packages_packages"."package_type" = 7
   AND "packages_packages"."name" = 'Solarbreeze'
   AND "packages_packages"."version" = '9.5.4'
 LIMIT 1

Explain plan:

Limit  (cost=0.41..3.44 rows=1 width=4) (actual time=0.948..0.949 rows=1 loops=1)
   Buffers: shared read=5
   I/O Timings: read=0.891
   ->  Index Only Scan using index_packages_on_project_id_name_version_unique_when_generic on public.packages_packages  (cost=0.41..3.44 rows=1 width=4) (actual time=0.947..0.947 rows=1 loops=1)
         Index Cond: ((packages_packages.project_id = 25155009) AND (packages_packages.name = 'Solarbreeze'::text) AND (packages_packages.version = '9.5.4'::text))
         Heap Fetches: 1
         Buffers: shared read=5
         I/O Timings: read=0.891

First

https://console.postgres.ai/shared/686f2956-f8a6-4a46-9198-ab3b747dc2fe

Query:

SELECT "packages_packages".*
  FROM "packages_packages"
 WHERE "packages_packages"."project_id" = 25155009
   AND "packages_packages"."package_type" = 7
   AND "packages_packages"."name" = 'Solarbreeze'
   AND "packages_packages"."version" = '9.5.4'
 ORDER BY "packages_packages"."id" ASC
 LIMIT 1

Explain plan:

Limit  (cost=3.44..3.45 rows=1 width=84) (actual time=8.189..8.191 rows=1 loops=1)
   Buffers: shared hit=3 read=4
   I/O Timings: read=8.089
   ->  Sort  (cost=3.44..3.45 rows=1 width=84) (actual time=8.187..8.188 rows=1 loops=1)
         Sort Key: packages_packages.id
         Sort Method: quicksort  Memory: 25kB
         Buffers: shared hit=3 read=4
         I/O Timings: read=8.089
         ->  Index Scan using index_packages_on_project_id_name_version_unique_when_generic on public.packages_packages  (cost=0.41..3.44 rows=1 width=84) (actual time=8.153..8.156 rows=1 loops=1)
               Index Cond: ((packages_packages.project_id = 25155009) AND ((packages_packages.name)::text = 'Solarbreeze'::text) AND ((packages_packages.version)::text = '9.5.4'::text))
               Buffers: shared read=4
               I/O Timings: read=8.089

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Ethan Reesor

Merge request reports