Skip to content

Add upstream endpoints on container virtual registry

Adie (she/her)requested to merge
548794-add-crud-for-upstream into master

What does this MR do and why?

In this MR, we add the following endpoints to manage the virtual registry upstreams.

🍋 API endpoints for VirtualRegistries::Container::Upstream

Route Notes
GET /groups/:id/-/virtual_registries/containers/upstreams Get upstreams for the given top level group
GET /virtual_registries/container/registries/:id/upstreams Get the upstreams of a given registry
POST /virtual_registries/container/registries/:id/upstreams Create an upstream for a given registry
GET /virtual_registries/container/upstreams/:id Get the upstream details
PATCH /virtual_registries/container/upstreams/:id Update the upstream details
DELETE /virtual_registries/container/upstreams/:id Delete an upstream

We add the above endpoints behind the feature flag container_virtual_registry and we also add a license check for this feature.

How to set up and validate locally

🍓 Prerequisites:

  1. Enable the feature flag 
Feature.enable(:container_virtual_registries)

  1. Have a **personal access token **ready. Here is a guide: https://docs.gitlab.com/user/profile/personal_access_tokens/#create-a-personal-access-token

  2. Have a group ID ready

Group.first.id

  1. Setup your GDK with an enterprise license (Premium). Follow these steps: https://gitlab.com/gitlab-org/customers-gitlab-com/-/blob/main/doc/setup/gitlab.md#adding-a-license-from-staging-customers-portal-to-your-gdk.

  2. cURL in your terminal or you may also use Postman (or similar)

🍓 Testing the endpoints:

1. Listing all virtual registries for the group: GET /groups/:id/-/virtual_registries/container/upstreams

curl --location 'http://gdk.test:3000/api/v4/groups/22/-/virtual_registries/container/upstreams' \
--header 'PRIVATE-TOKEN: glpat-xxxxx'

Result will look like, depending on what is in your database:

[
    {
        "id": 4,
        "name": "DockerHub Test",
        "description": "Test DockerHub upstream - failcase",
        "group_id": 22,
        "url": "https://registry-1.docker.io",
        "username": "jagodie33",
        "cache_validity_hours": 24,
        "created_at": "2025-08-22T09:30:39.474Z",
        "updated_at": "2025-08-22T09:30:39.474Z"
    }
]

2. Get the upstreams of a given registry: GET /virtual_registries/container/registries/:id/upstreams

curl --location 'http://gdk.test:3000/api/v4/virtual_registries/container/registries/3/upstreams' \
--header 'PRIVATE-TOKEN: glpat-xxxxx'

Result:

[
    {
        "id": 8,
        "name": "upstream1",
        "description": null,
        "group_id": 22,
        "url": "https://sample.com",
        "username": "testuser",
        "cache_validity_hours": 24,
        "created_at": "2025-09-20T10:15:40.258Z",
        "updated_at": "2025-09-20T10:15:40.258Z",
        "registry_upstream": {
            "id": 9,
            "position": 1,
            "registry_id": 3
        }
    }
]

3. Creating a new virtual registry: POST /virtual_registries/container/registries/:registry_id/upstreams

Feel free to update the parameters.

curl --location 'http://gdk.test:3000/api/v4/virtual_registries/container/registries/3/upstreams' \
--header 'PRIVATE-TOKEN: glpat-xxxxx' \
--form 'url="https://sample.com"' \
--form 'username="testuser"' \
--form 'password="testpass"' \
--form 'name="upstream1"'

Result will look something like:

{
    "id": 8,
    "name": "upstream1",
    "description": null,
    "group_id": 22,
    "url": "https://sample.com",
    "username": "testuser",
    "cache_validity_hours": 24,
    "created_at": "2025-09-20T10:15:40.258Z",
    "updated_at": "2025-09-20T10:15:40.258Z",
    "registry_upstream": {
        "id": 9,
        "position": 1,
        "registry_id": 3
    }
}

You can try the GET request again from (1) and you should see this newly created upstream as a part of the list.

4. Get the details of a virtual registry upstream: GET /virtual_registries/container/upstreams/:id 

curl --location 'http://gdk.test:3000/api/v4/virtual_registries/container/upstreams/8' \
--header 'PRIVATE-TOKEN: glpat-xxxxx'

The result would look like:

{
    "id": 8,
    "name": "upstream1",
    "description": null,
    "group_id": 22,
    "url": "https://sample.com",
    "username": "testuser",
    "cache_validity_hours": 24,
    "created_at": "2025-09-20T10:15:40.258Z",
    "updated_at": "2025-09-20T10:15:40.258Z",
    "registry_upstreams": [
        {
            "id": 9,
            "position": 1,
            "registry_id": 3
        }
    ]
}

5. Updating a virtual registry upstream: PATCH /virtual_registries/container/upstreams/:id

curl --location --request PATCH 'http://gdk.test:3000/api/v4/virtual_registries/container/upstreams/8' \
--header 'PRIVATE-TOKEN: glpat-xxxxx' \
--form 'name="upstream1"'

Result would be a 200 if successful.

You can try doing the GET request in (3) and should see the updated values.

6. Deleting a virtual registry: DELETE /virtual_registries/container/upstreams/:id

curl --location --request DELETE 'http://gdk.test:3000/api/v4/virtual_registries/container/upstreams/8' \
--header 'PRIVATE-TOKEN: glpat-xxxxx'

Result would be a 204 No Content if successful.

You can try again the the request in (1) to list all the virtual registries of the group and the deleted virtual registry would no longer be a part of it.

🍎 Database Query Plans

  1. 👉 Inserting an upstream
SQL Query 
INSERT INTO "virtual_registries_container_upstreams" ("group_id", "created_at", "updated_at", "username", "password", "url", "name")
    VALUES (22, '2025-09-20 13:18:49.227018', '2025-09-20 13:18:49.227018', '{"p":"26ArqW6I","h":{"iv":"I85F5wMkNQ5orxTV","at":"F1Mva8QLHleaj0mM0OdUZg==","i":"YTdjNg=="}}', '{"p":"TBtmevy4","h":{"iv":"+DiRCa3YNEJjENEv","at":"IW0JmKgvIPyMILzKyy4qag==","i":"YTdjNg=="}}', 'https://aa.com', 'test')
RETURNING
    "id"
Execution Plan 
 ModifyTable on public.virtual_registries_container_upstreams  (cost=0.00..0.01 rows=1 width=194) (actual time=0.673..0.675 rows=1 loops=1)
   Buffers: shared hit=78 read=4 dirtied=6 written=3
   WAL: records=7 fpi=0 bytes=811
   I/O Timings: read=0.119 write=0.060
   ->  Result  (cost=0.00..0.01 rows=1 width=194) (actual time=0.166..0.166 rows=1 loops=1)
         Buffers: shared hit=13 read=1 dirtied=1
         WAL: records=1 fpi=0 bytes=99
         I/O Timings: read=0.051 write=0.000
Trigger RI_ConstraintTrigger_c_1067341121 for constraint fk_rails_c97afd8bbd: time=2.691 calls=1
Settings: effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', work_mem = '100MB'

Details and visualization: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/43631/commands/133259

  1. 👉 Updating a registry upstream
SQL Query 
UPDATE
    "virtual_registries_container_upstreams"
SET
    "group_id" = 22,
    "updated_at" = '2025-09-20 13:33:35.651236',
    "name" = 'asdf'
WHERE
    "virtual_registries_container_upstreams"."id" = 1
Execution Plan 
ModifyTable on public.virtual_registries_container_upstreams  (cost=0.14..3.16 rows=0 width=0) (actual time=0.022..0.023 rows=0 loops=1)
   Buffers: shared hit=6
   I/O Timings: read=0.000 write=0.000
   ->  Index Scan using virtual_registries_container_upstreams_pkey on public.virtual_registries_container_upstreams  (cost=0.14..3.16 rows=1 width=54) (actual time=0.021..0.021 rows=0 loops=1)
         Index Cond: (virtual_registries_container_upstreams.id = 1)
         Buffers: shared hit=6
         I/O Timings: read=0.000 write=0.000
Settings: work_mem = '100MB', effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4'

Details and visualization: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/43631/commands/133260

  1. 👉 Deleting a registry upstream
SQL Query 
DELETE FROM "virtual_registries_container_upstreams"
WHERE "virtual_registries_container_upstreams"."id" = 1
Execution Plan 
 ModifyTable on public.virtual_registries_container_upstreams  (cost=0.14..3.16 rows=0 width=0) (actual time=0.017..0.018 rows=0 loops=1)
   Buffers: shared hit=4
   I/O Timings: read=0.000 write=0.000
   ->  Index Scan using virtual_registries_container_upstreams_pkey on public.virtual_registries_container_upstreams  (cost=0.14..3.16 rows=1 width=6) (actual time=0.016..0.016 rows=0 loops=1)
         Index Cond: (virtual_registries_container_upstreams.id = 1)
         Buffers: shared hit=4
         I/O Timings: read=0.000 write=0.000
Settings: effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', work_mem = '100MB'

Details and visualization: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/43631/commands/133263

  1. 👉 Fetching upstreams of the group (max 5 upstreams per registry, and max 5 registry per group)
SQL Query 
SELECT
    "virtual_registries_container_upstreams".*
FROM
    "virtual_registries_container_upstreams"
WHERE
    "virtual_registries_container_upstreams"."group_id" = 22
Execution Plan 
Index Scan using virtual_registries_container_upstreams_on_group_id on public.virtual_registries_container_upstreams  (cost=0.14..3.16 rows=1 width=194) (actual time=0.027..0.027 rows=0 loops=1)
   Index Cond: (virtual_registries_container_upstreams.group_id = 22)
   Buffers: shared hit=6
   I/O Timings: read=0.000 write=0.000
Settings: work_mem = '100MB', effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4'

Details and visualization:: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/43631/commands/133261

  1. 👉 Fetching upstreams of a virtual registry (max 5 upstreams per registry)
SQL Query 
SELECT
    "virtual_registries_container_upstreams"."id" AS t0_r0,
    "virtual_registries_container_upstreams"."group_id" AS t0_r1,
    "virtual_registries_container_upstreams"."created_at" AS t0_r2,
    "virtual_registries_container_upstreams"."updated_at" AS t0_r3,
    "virtual_registries_container_upstreams"."cache_validity_hours" AS t0_r4,
    "virtual_registries_container_upstreams"."username" AS t0_r5,
    "virtual_registries_container_upstreams"."password" AS t0_r6,
    "virtual_registries_container_upstreams"."url" AS t0_r7,
    "virtual_registries_container_upstreams"."name" AS t0_r8,
    "virtual_registries_container_upstreams"."description" AS t0_r9,
    "registry_upstreams"."id" AS t1_r0,
    "registry_upstreams"."group_id" AS t1_r1,
    "registry_upstreams"."registry_id" AS t1_r2,
    "registry_upstreams"."upstream_id" AS t1_r3,
    "registry_upstreams"."created_at" AS t1_r4,
    "registry_upstreams"."updated_at" AS t1_r5,
    "registry_upstreams"."position" AS t1_r6
FROM
    "virtual_registries_container_upstreams"
    LEFT OUTER JOIN "virtual_registries_container_registry_upstreams" "registry_upstreams" ON "registry_upstreams"."upstream_id" = "virtual_registries_container_upstreams"."id"
WHERE
    "registry_upstreams"."registry_id" = 5
    AND "virtual_registries_container_upstreams"."id" = 9
Execution Plan 
 Nested Loop  (cost=0.28..6.33 rows=1 width=244) (actual time=0.018..0.019 rows=0 loops=1)
   Buffers: shared hit=4
   I/O Timings: read=0.000 write=0.000
   ->  Index Scan using virtual_registries_container_upstreams_pkey on public.virtual_registries_container_upstreams  (cost=0.14..3.16 rows=1 width=194) (actual time=0.017..0.017 rows=0 loops=1)
         Index Cond: (virtual_registries_container_upstreams.id = 9)
         Buffers: shared hit=4
         I/O Timings: read=0.000 write=0.000
   ->  Index Scan using constraint_vreg_container_reg_upst_on_unique_reg_pos on public.virtual_registries_container_registry_upstreams registry_upstreams  (cost=0.14..3.16 rows=1 width=50) (actual time=0.000..0.000 rows=0 loops=0)
         Index Cond: (registry_upstreams.registry_id = 5)
         Filter: (registry_upstreams.upstream_id = 9)
         Rows Removed by Filter: 0
         I/O Timings: read=0.000 write=0.000
Settings: seq_page_cost = '4', work_mem = '100MB', effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5'

Details and visualization:: https://postgres.ai/console/gitlab/gitlab-production-main/sessions/43631/commands/133262

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #548794 (closed)

Edited by Adie (she/her)

Merge request reports