Add latest package revision endpoint (!187678) · Merge requests · GitLab.org / GitLab

Please check this box if this contribution uses AI-generated content (including content generated by GitLab Duo features) as outlined in the GitLab DCO & CLA. As a benefit of being a GitLab Community Contributor, you receive complimentary access to GitLab Duo.

What does this MR do and why?

Add latest package revision endpoint

Add a new API endpoint to get the latest package revision for a specific recipe revision and package reference in the Conan package registry.

Add new GET endpoint /api/v4/projects/:id/packages/conan/v2/conans/:package_name/:package_version/:package_username/:package_channel/revisions/:recipe_revision/packages/:conan_package_reference/latest
Add new scopes order_by_id_desc and by_recipe_revision_and_package_reference to PackageRevision model
Rename RecipeRevision entity to Revision to make it reusable for both recipe and package revisions
Add comprehensive tests for new endpoint and model scopes

Changelog: added

References

#519741 (closed): Conan V2 follow up issue

Screenshots or screen recordings

No UI changes

How to set up and validate locally

Switch the conan_package_revisions_support feature flag on.

For GDK, open rails console (docs):

gdk rails console

Enable the feature flag:

# Enable
Feature.enable(:conan_package_revisions_support)

# Verify
Feature.enabled?(:conan_package_revisions_support)

Get Conan Authentication Token

First, generate a base64-encoded Basic Auth token:

# Replace with your username and PAT
echo -n "USERNAME:glpat-YOUR-TOKEN"|base64

Then use the output to get a Conan JWT token:

curl --request GET \
--url 'http://localhost:3000/api/v4/projects/<project_id>/packages/conan/v1/users/authenticate' \
--header 'Authorization: Basic <YOUR-BASE64-TOKEN>'

Create a test package file

# Create a simple package file
echo "info" > conaninfo.txt

Upload the same file with different package revisions

Upload with first package revision (older SHA1):

curl --request PUT \
  --form 'file=@conaninfo.txt' \
  'http://localhost:3000/api/v4/projects/<project_id>/packages/conan/v2/conans/test-package/1.0.0/user/stable/revisions/75151329520e7685dcf5da49ded2fec0/packages/103f6067a947f366ef91fc1b7da351c588d1827f/revisions/3bdd2d8c8e76c876ebd1ac0469a4e72c/files/conaninfo.txt' \
  --header 'Authorization: Bearer <YOUR-CONAN-JWT-TOKEN>'

Upload same file with second package revision (newer SHA1):

curl --request PUT \
  --form 'file=@conaninfo.txt' \
  'http://localhost:3000/api/v4/projects/<project_id>/packages/conan/v2/conans/test-package/1.0.0/user/stable/revisions/75151329520e7685dcf5da49ded2fec0/packages/103f6067a947f366ef91fc1b7da351c588d1827f/revisions/5f9c4ab08cac7457e9111a30e4664920/files/conaninfo.txt' \
  --header 'Authorization: Bearer <YOUR-CONAN-JWT-TOKEN>'

Test the new latest package revision endpoint

Call the endpoint to get the latest package revision:

curl --request GET \
  'http://localhost:3000/api/v4/projects/<project_id>/packages/conan/v2/conans/test-package/1.0.0/user/stable/revisions/75151329520e7685dcf5da49ded2fec0/packages/103f6067a947f366ef91fc1b7da351c588d1827f/latest' \
  --header 'Authorization: Bearer <YOUR-CONAN-JWT-TOKEN>'

Expected response (should return the most recent revision):

{
  "revision": "5f9c4ab08cac7457e9111a30e4664920",
  "time": "<Time of upload in iso-8601 format: 2025-03-04T12:46:18.844Z>"
}

💾 Database Review

`Query 1: Packages::Conan::Package Load`

NOTE: This is the same as Query 1 reviewed in !187107 (merged)

pg.ai setup

reset

Query

SELECT
    "packages_packages".*
FROM
    "packages_packages"
    INNER JOIN "packages_conan_metadata" ON "packages_conan_metadata"."package_id" = "packages_packages"."id"
WHERE
    "packages_packages"."package_type" = 3
    AND "packages_packages"."project_id" = 36299920
    AND "packages_packages"."name" = 'packageReferenceTest'
    AND "packages_packages"."version" = '1.2.4'
    AND "packages_conan_metadata"."package_username" = 'issue-reproduce+conan'
    AND "packages_conan_metadata"."package_channel" = 'stable'
    AND "packages_packages"."status" != 4
ORDER BY
    "packages_packages"."created_at" DESC
LIMIT
    1

Explain plan

 Limit  (cost=10.06..10.07 rows=1 width=141) (actual time=14.361..14.363 rows=1 loops=1)
   Buffers: shared hit=9 read=6 dirtied=1
   WAL: records=2 fpi=1 bytes=8254
   I/O Timings: read=14.074 write=0.000
   ->  Sort  (cost=10.06..10.07 rows=1 width=141) (actual time=14.360..14.361 rows=1 loops=1)
         Sort Key: packages_packages.created_at DESC
         Sort Method: quicksort  Memory: 25kB
         Buffers: shared hit=9 read=6 dirtied=1
         WAL: records=2 fpi=1 bytes=8254
         I/O Timings: read=14.074 write=0.000
         ->  Nested Loop  (cost=0.98..10.05 rows=1 width=141) (actual time=14.325..14.329 rows=1 loops=1)
               Buffers: shared hit=6 read=6 dirtied=1
               WAL: records=2 fpi=1 bytes=8254
               I/O Timings: read=14.074 write=0.000
               ->  Index Scan using idx_packages_packages_on_project_id_name_version_package_type on public.packages_packages  (cost=0.56..3.59 rows=1 width=141) (actual time=13.078..13.080 rows=1 loops=1)
                     Index Cond: ((packages_packages.project_id = 36299920) AND ((packages_packages.name)::text = 'packageReferenceTest'::text) AND ((packages_packages.version)::text = '1.2.4'::text) AND (packages_packages.package_type = 3))
                     Filter: (packages_packages.status <> 4)
                     Rows Removed by Filter: 0
                     Buffers: shared hit=3 read=5 dirtied=1
                     WAL: records=2 fpi=1 bytes=8254
                     I/O Timings: read=12.863 write=0.000
               ->  Index Only Scan using index_packages_conan_metadata_on_package_id_username_channel on public.packages_conan_metadata  (cost=0.42..3.44 rows=1 width=8) (actual time=1.241..1.241 rows=1 loops=1)
                     Index Cond: ((packages_conan_metadata.package_id = packages_packages.id) AND (packages_conan_metadata.package_username = 'issue-reproduce+conan'::text) AND (packages_conan_metadata.package_channel = 'stable'::text))
                     Heap Fetches: 0
                     Buffers: shared hit=3 read=1
                     I/O Timings: read=1.211 write=0.000
Settings: effective_cache_size = '472585MB', jit = 'off', random_page_cost = '1.5', seq_page_cost = '4', work_mem = '100MB'

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/38346/commands/117692

`Query 2: Packages::Conan::PackageRevision Load`

pg.ai setup

reset

Query

SELECT
    "packages_conan_package_revisions".*
FROM
    "packages_conan_package_revisions"
    INNER JOIN "packages_conan_package_references" ON "packages_conan_package_references"."id" = "packages_conan_package_revisions"."package_reference_id"
    INNER JOIN "packages_conan_recipe_revisions" ON "packages_conan_recipe_revisions"."id" = "packages_conan_package_references"."recipe_revision_id"
WHERE
    "packages_conan_package_revisions"."package_id" = 36923910
    AND "packages_conan_recipe_revisions"."revision" = '\x75151329520e7685dcf5da49ded2fec0'
    AND "packages_conan_package_references"."reference" = '\x103f6067a947f366ef91fc1b7da351c588d1827f'
ORDER BY
    "packages_conan_package_revisions"."id" DESC
LIMIT
    1

Explain plan

 Limit  (cost=3.50..3.51 rows=1 width=80) (actual time=0.025..0.026 rows=0 loops=1)
   Buffers: shared hit=3
   I/O Timings: read=0.000 write=0.000
   ->  Sort  (cost=3.50..3.51 rows=1 width=80) (actual time=0.024..0.025 rows=0 loops=1)
         Sort Key: packages_conan_package_revisions.id DESC
         Sort Method: quicksort  Memory: 25kB
         Buffers: shared hit=3
         I/O Timings: read=0.000 write=0.000
         ->  Nested Loop  (cost=0.42..3.49 rows=1 width=80) (actual time=0.006..0.006 rows=0 loops=1)
               I/O Timings: read=0.000 write=0.000
               ->  Nested Loop  (cost=0.42..3.46 rows=3 width=8) (actual time=0.005..0.006 rows=0 loops=1)
                     I/O Timings: read=0.000 write=0.000
                     ->  Seq Scan on public.packages_conan_recipe_revisions  (cost=0.00..0.00 rows=1 width=8) (actual time=0.005..0.005 rows=0 loops=1)
                           Filter: (packages_conan_recipe_revisions.revision = '\x75151329520e7685dcf5da49ded2fec0'::bytea)
                           Rows Removed by Filter: 0
                           I/O Timings: read=0.000 write=0.000
                     ->  Index Scan using index_packages_conan_package_references_on_recipe_revision_id on public.packages_conan_package_references  (cost=0.42..3.45 rows=1 width=16) (actual time=0.000..0.000 rows=0 loops=0)
                           Index Cond: (packages_conan_package_references.recipe_revision_id = packages_conan_recipe_revisions.id)
                           Filter: (packages_conan_package_references.reference = '\x103f6067a947f366ef91fc1b7da351c588d1827f'::bytea)
                           Rows Removed by Filter: 0
                           I/O Timings: read=0.000 write=0.000
               ->  Seq Scan on public.packages_conan_package_revisions  (cost=0.00..0.00 rows=1 width=80) (actual time=0.000..0.000 rows=0 loops=0)
                     Filter: (packages_conan_package_revisions.package_id = 36923910)
                     Rows Removed by Filter: 0
                     I/O Timings: read=0.000 write=0.000
Settings: jit = 'off', effective_cache_size = '472585MB', seq_page_cost = '4', work_mem = '100MB', random_page_cost = '1.5'

https://console.postgres.ai/gitlab/gitlab-production-main/sessions/38485/commands/118348

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

/cc @mbo5be

Related to #519741 (closed)

Edited Apr 15, 2025 by Radamanthus Batnag