Support NuGet version normalization
What does this MR do and why?
Problem
Suppose we have a NuGet package named Hello
in our GitLab NuGet Repository with version 1.0.7.0
. If we tried to pull it using NuGet 3.4+, you would get the following error:
Package 'Hello.1.0.7' is not found on source 'https://example.gitlab.com/api/v4/projects/<your_project_id>/packages/nuget/index.json'.
Notice the version in the error message: it's different than the package's version stored in GitLab's NuGet Repository. The reason is that NuGet 3.4+ normalizes the version number and then sends the normalized string to the NuGet Repository. We try to search for the package using the received normalized version and we don't find the package. Makes sense.
Solution
In this MR, the following solution has been introduced:
- Create a new column
normalized_version
in thepackages_nuget_metadata
table - Add Packages::Nuget::NormalizeVersionService to apply normalization rules on the version string.
- When a new package is created, we save the normalized version in
packages_nuget_metadata
'snormalized_version
column using abefore_validation
callback in thePackages::Nuget::Metadatum
model. - Update
Packages::Nuget::PackageFinder
to search for packages usingLOWER(version)
orpackages_nuget_metadata
'snormalized_version
. - Get rid of
ILIKE
fuzzing matching inPackages::Nuget::PackageFinder
in favor of exact matching to enhance the performance. NuGet client always sends the package's full name so exact matching would suffice. - In a subsequent MR, I will introduce a background migration to backfill the
normalized_version
for the existing NuGet Repository packages. - Changes are behind the
nuget_normalized_version
feature flag.
How to set up and validate locally
-
Ensure you have the NuGet CLI installed (see nuget docs for links to installation pages).
-
In a new directory, run
nuget spec
. -
Update the version in the
Package.nuspec
file to1.0.7.0
. -
Run
nuget pack
in the same directory, you should find a generated file namedPackage.1.0.7.nupkg
. -
Notice the NuGet CLI automatically normalized the version to
1.0.7
, we need to update the version manually so that we can push the package with the unnormalized version1.0.7.0
:- Run
unzip Package.1.0.7.nupkg
, and allow the.nuspec
file to be replaced. - Edit the
.nuspec
file and set the version to1.0.7.0
- Compress/zip all of the files in the folder except for
Package.1.0.7.nupkg
. - Rename the zip archive
Package.1.0.7.0.nupkg
.
- Run
-
Add a GitLab project as your NuGet source:
nuget source Add -Name localhost -Source "http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/index.json" -UserName <gitlab_username> -Password <personal_access_token>
-
In rails console, enable the
nuget_normalized_version
feature flag before publishing the package:
Feature.enable(:nuget_normalized_version, Project.find(<project_id>))
-
Push the package to your project:
nuget push Package.1.0.7.0.nupkg.zip -Source localhost
-
In the UI, navigate to the project, and go to
Packages & Registries -> Package Registry
. You should see the NuGet package published with version1.0.7.0
. -
Create packages.config file in the directory of the published package:
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="Package" version="1.0.7.0" />
</packages>
- Execute the command:
nuget restore -PackagesDirectory . -Source localhost
- The package should be pulled successfully.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Database analysis
Related to #273697 (closed) & #270976 (closed)
Merge request reports
Activity
changed milestone to %16.1
assigned to @mkhalifa3
- A deleted user
added database databasereview pending labels
1 Warning 027bf023: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend Kasia Misirli (
@Kasia_Misirli
) (UTC+2)Jessie Young (
@jessieay
) (UTC+8)database Max Woolf (
@mwoolf
) (UTC+1)Dylan Griffith (
@DylanGriffith
) (UTC+10)~"migration" No reviewer available No maintainer available To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger- A deleted user
added Data WarehouseImpact Check label
added bugfunctional label
- Resolved by Mayra Cabrera
Database migrations (on the main database)
1 Warnings 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum had a query that exceeded
timing guidelines. Run time should not exceed 100ms, but it was 317.39ms. Please consider possible
options to improve the query performance.ALTER TABLE packages_nuget_metadata VALIDATE
CONSTRAINT check_9973c0cc33Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum Regular 3.6 s +0.00 B 20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion Regular 2.6 s +8.97 MiB 20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages Regular 56.6 s +48.27 MiB Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 8 0.1 seconds - 1 second 2 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 1 5 minutes + 0 Migration: 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum- Type: Regular
- Duration: 3.6 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 317.4 ms 317.4 ms 317.4 ms 0 ALTER TABLE packages_nuget_metadata VALIDATE CONSTRAINT check_9973c0cc33
1 18.7 ms 18.7 ms 18.7 ms 0 ALTER TABLE packages_nuget_metadata ADD CONSTRAINT check_9973c0cc33 CHECK ( char_length(normalized_version) <= 255 ) NOT VALID
1 6.9 ms 6.9 ms 6.9 ms 0 ALTER TABLE "packages_nuget_metadata" ADD "normalized_version" text
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddNormalizedVersionToPackagesNugetMetadatum
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 4 0.1 seconds - 1 second 1 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion
- Type: Regular
- Duration: 2.6 s
- Database size change: +8.97 MiB
Calls Total Time Max Time Mean Time Rows Query 1 346.4 ms 346.4 ms 346.4 ms 0 CREATE INDEX CONCURRENTLY "idx_packages_nuget_metadata_on_package_id_and_normalize_version" ON "packages_nuget_metadata" (package_id, normalized_version)
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 1 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages
- Type: Regular
- Duration: 56.6 s
- Database size change: +48.27 MiB
Calls Total Time Max Time Mean Time Rows Query 1 53542.9 ms 53542.9 ms 53542.9 ms 0 CREATE INDEX CONCURRENTLY "index_packages_packages_on_project_id_and_lower_name" ON "packages_packages" (project_id, LOWER(name))
WHERE package_type = 42 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 2 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 1 5 minutes + 0 Other information
No other migrations pending on GitLab.com
Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-2153889-10709102-main
2023-07-26T20:54:45Z 2023-07-26T20:11:34Z 2023-07-27 09:01:32 +0000 database-testing-2153889-10709102-ci
2023-07-26T20:54:45Z 2023-07-26T16:46:10Z 2023-07-27 09:01:32 +0000 Database migrations (on the ci database)
Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum Regular 3.8 s +0.00 B 20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion Regular 3.1 s +8.00 KiB [note] 20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages Regular 3.9 s +8.00 KiB [note] Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 11 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum
- Type: Regular
- Duration: 3.8 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 10.3 ms 10.3 ms 10.3 ms 0 ALTER TABLE packages_nuget_metadata ADD CONSTRAINT check_9973c0cc33 CHECK ( char_length(normalized_version) <= 255 ) NOT VALID
1 7.4 ms 7.4 ms 7.4 ms 0 ALTER TABLE "packages_nuget_metadata" ADD "normalized_version" text
1 1.6 ms 1.6 ms 1.6 ms 0 ALTER TABLE packages_nuget_metadata VALIDATE CONSTRAINT check_9973c0cc33
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddNormalizedVersionToPackagesNugetMetadatum
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 5 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion
- Type: Regular
- Duration: 3.1 s
- Database size change: +8.00 KiB [note]
Calls Total Time Max Time Mean Time Rows Query 1 18.7 ms 18.7 ms 18.7 ms 0 CREATE INDEX CONCURRENTLY "idx_packages_nuget_metadata_on_package_id_and_normalize_version" ON "packages_nuget_metadata" (package_id, normalized_version)
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 3 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages
- Type: Regular
- Duration: 3.9 s
- Database size change: +8.00 KiB [note]
Calls Total Time Max Time Mean Time Rows Query 1 42.0 ms 42.0 ms 42.0 ms 0 CREATE INDEX CONCURRENTLY "index_packages_packages_on_project_id_and_lower_name" ON "packages_packages" (project_id, LOWER(name))
WHERE package_type = 42 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 3 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Other information
No other migrations pending on GitLab.com
Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-2153889-10709102-main
2023-07-26T20:54:45Z 2023-07-26T20:11:34Z 2023-07-27 09:01:32 +0000 database-testing-2153889-10709102-ci
2023-07-26T20:54:45Z 2023-07-26T16:46:10Z 2023-07-27 09:01:32 +0000
Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic
- A deleted user
added database-testing-automation label
- Resolved by Steve Abrams
Hi @ck3g
a database review is needed here, would you mind doing it if you have the capacity? Thank youHi @dmeshcharakou
would you mind doing the backend review if you have the capacity? Thank you
requested review from @ck3g
requested review from @dmeshcharakou
removed [deprecated] Accepting merge requests label
mentioned in issue #273697 (closed)