Read bundled catalog component documents from object storage

What

Adds Gitlab::Ci::Catalog::Bundle::Reader, which reads a compiled component bundle from the cell's own object storage, plus the finders it needs to locate one by natural key: server_fqdn, full_path, semver and component name.

Why

On a Cell, a customer cannot reach the project that publishes a GitLab-maintained CI component, so the component cannot be fetched from its source. Each version is instead compiled into a self-contained bundle and stored on the cell. This is the read side of that work.

There is no caller yet. #600464 (closed) adds one during pipeline creation.

References

Database review

Three read-only point lookups. Each predicate is fully covered by a unique btree index, so every condition is an index condition rather than a filter.

1. Resolve the resource by natural key

SELECT "catalog_bundled_resources".* FROM "catalog_bundled_resources"
WHERE "server_fqdn" = 'gitlab.com' AND "full_path" = 'gitlab-org/components/opentofu'
ORDER BY "id" ASC LIMIT 1
Limit  (cost=2.18..2.18 rows=1 width=192)
  ->  Sort  (cost=2.18..2.18 rows=1 width=192)
        Sort Key: id
        ->  Index Scan using index_catalog_bundled_resources_on_server_fqdn_and_full_path on catalog_bundled_resources  (cost=0.15..2.17 rows=1 width=192)
              Index Cond: ((server_fqdn = 'gitlab.com'::text) AND (full_path = 'gitlab-org/components/opentofu'::text))

The Sort comes from .first and runs over at most one row, since the index is unique on both filtered columns.

2. Resolve the version by semver

SELECT "catalog_bundled_resource_versions".<columns> FROM "catalog_bundled_resource_versions"
WHERE "catalog_bundled_resource_id" = 1 AND "semver_major" = 1 AND "semver_minor" = 2
  AND "semver_patch" = 3 AND "semver_prerelease" IS NULL
LIMIT 1
Limit  (cost=0.15..2.17 rows=1 width=91)
  ->  Index Scan using idx_catalog_bundled_versions_on_bundled_resource_and_semver on catalog_bundled_resource_versions  (cost=0.15..2.17 rows=1 width=91)
        Index Cond: ((catalog_bundled_resource_id = 1) AND (semver_major = 1) AND (semver_minor = 2) AND (semver_patch = 3) AND (semver_prerelease IS NULL))

The index is NULLS NOT DISTINCT, which is what keeps released versions indexable and unique with a NULL prerelease.

3. Resolve the component by name within the version

SELECT "catalog_bundled_resource_components".* FROM "catalog_bundled_resource_components"
WHERE "catalog_bundled_version_id" = 1 AND "name" = 'opentofu'
LIMIT 1
Limit  (cost=0.15..2.17 rows=1 width=130)
  ->  Index Scan using idx_catalog_bundled_components_on_version_and_name on catalog_bundled_resource_components  (cost=0.15..2.17 rows=1 width=130)
        Index Cond: ((catalog_bundled_version_id = 1) AND (name = 'opentofu'::text))

On the cost numbers. All three tables are empty, locally and in production, because nothing writes them until the publish-time compile in #600454 (closed). So the costs carry no information and database lab would not add any. The index coverage is the part that holds as the tables grow.

Edited by Laura Montemayor

Merge request reports

Loading
Loading