Support NuGet version normalization

What does this MR do and why?

Problem

Suppose we have a NuGet package named Hello in our GitLab NuGet Repository with version 1.0.7.0. If we tried to pull it using NuGet 3.4+, you would get the following error:

Package 'Hello.1.0.7' is not found on source 'https://example.gitlab.com/api/v4/projects/<your_project_id>/packages/nuget/index.json'.

Notice the version in the error message: it's different than the package's version stored in GitLab's NuGet Repository. The reason is that NuGet 3.4+ normalizes the version number and then sends the normalized string to the NuGet Repository. We try to search for the package using the received normalized version and we don't find the package. Makes sense.

Solution

In this MR, the following solution has been introduced:

• Create a new column normalized_version in the packages_nuget_metadata table
• Add Packages::Nuget::NormalizeVersionService to apply normalization rules on the version string.
• When a new package is created, we save the normalized version in packages_nuget_metadata 's normalized_version column using a before_validation callback in the Packages::Nuget::Metadatum model.
• Update Packages::Nuget::PackageFinder to search for packages using LOWER(version) or packages_nuget_metadata 's normalized_version.
• Get rid of ILIKE fuzzing matching in Packages::Nuget::PackageFinder in favor of exact matching to enhance the performance. NuGet client always sends the package's full name so exact matching would suffice.
• In a subsequent MR, I will introduce a background migration to backfill the normalized_version for the existing NuGet Repository packages.
• Changes are behind the nuget_normalized_version feature flag.

How to set up and validate locally

1. Ensure you have the NuGet CLI installed (see nuget docs for links to installation pages).

2. In a new directory, run nuget spec.

3. Update the version in the Package.nuspec file to 1.0.7.0.

4. Run nuget pack in the same directory, you should find a generated file named Package.1.0.7.nupkg.

5. Notice the NuGet CLI automatically normalized the version to 1.0.7, we need to update the version manually so that we can push the package with the unnormalized version 1.0.7.0:

   1. Run unzip Package.1.0.7.nupkg, and allow the .nuspec file to be replaced.
   2. Edit the .nuspec file and set the version to 1.0.7.0
   3. Compress/zip all of the files in the folder except for Package.1.0.7.nupkg.
   4. Rename the zip archive Package.1.0.7.0.nupkg.

6. Add a GitLab project as your NuGet source:

   nuget source Add -Name localhost -Source "http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/index.json" -UserName <gitlab_username> -Password <personal_access_token>

7. In rails console, enable the nuget_normalized_version feature flag before publishing the package:

Feature.enable(:nuget_normalized_version, Project.find(<project_id>))

8. Push the package to your project:

   nuget push Package.1.0.7.0.nupkg.zip -Source localhost

9. In the UI, navigate to the project, and go to Packages & Registries -> Package Registry. You should see the NuGet package published with version 1.0.7.0.

10. Create packages.config file in the directory of the published package:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Package" version="1.0.7.0" />
</packages>

11. Execute the command: nuget restore -PackagesDirectory . -Source localhost
12. The package should be pulled successfully.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Database analysis

• !121260 (comment 1479631431)
• !121260 (comment 1479631469)

Related to #273697 (closed) & #270976 (closed)

changed milestone to %16.1

added Category:Package Registry NuGet Repository SLOMissed [deprecated] Accepting merge requests backend customer devopspackage grouppackage registry priority2 ruby sectionops severity2 typebug workflowin dev labels

assigned to @mkhalifa3

added 1 commit

• 9a86ace0 - Add index on normalize_version function

Compare with previous version

added database databasereview pending labels

	1 Warning
	027bf023: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Kasia Misirli (@Kasia_Misirli) (UTC+2)	Jessie Young (@jessieay) (UTC+8)
database	Max Woolf (@mwoolf) (UTC+1)	Dylan Griffith (@DylanGriffith) (UTC+10)
~"migration"	No reviewer available	No maintainer available

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added Data WarehouseImpact Check label

added 1 commit

• c1554b8f - Fix spec title

Compare with previous version

added 1 commit

• 43f7bd8c - Fix PackageFinder specs

Compare with previous version

added bugfunctional label

changed the description

Database migrations (on the main database)

	1 Warnings
	20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum had a query that exceeded timing guidelines. Run time should not exceed 100ms, but it was 317.39ms. Please consider possible options to improve the query performance. ALTER TABLE packages_nuget_metadata VALIDATE CONSTRAINT check_9973c0cc33

Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

Migration	Type	Total runtime	Result	DB size change
20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum	Regular	3.6 s		+0.00 B
20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion	Regular	2.6 s		+8.97 MiB
20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages	Regular	56.6 s		+48.27 MiB

Runtime Histogram for all migrations

Query Runtime	Count
0 seconds - 0.01 seconds	0
0.01 seconds - 0.1 seconds	8
0.1 seconds - 1 second	2
1 second - 5 seconds	0
5 seconds - 15 seconds	0
15 seconds - 5 minutes	1
5 minutes +	0

Migration: 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum

• Type: Regular
• Duration: 3.6 s
• Database size change: +0.00 B

Calls	Total Time	Max Time	Mean Time	Rows	Query
1	317.4 ms	317.4 ms	317.4 ms	0	ALTER TABLE packages_nuget_metadata VALIDATE CONSTRAINT check_9973c0cc33
1	18.7 ms	18.7 ms	18.7 ms	0	ALTER TABLE packages_nuget_metadata ADD CONSTRAINT check_9973c0cc33 CHECK ( char_length(normalized_version) <= 255 ) NOT VALID
1	6.9 ms	6.9 ms	6.9 ms	0	ALTER TABLE "packages_nuget_metadata" ADD "normalized_version" text
2	0.0 ms	0.0 ms	0.0 ms	2	SELECT pg_backend_pid()

Histogram for AddNormalizedVersionToPackagesNugetMetadatum

Query Runtime	Count
0 seconds - 0.01 seconds	0
0.01 seconds - 0.1 seconds	4
0.1 seconds - 1 second	1
1 second - 5 seconds	0
5 seconds - 15 seconds	0
15 seconds - 5 minutes	0
5 minutes +	0

Migration: 20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion

• Type: Regular
• Duration: 2.6 s
• Database size change: +8.97 MiB

Calls	Total Time	Max Time	Mean Time	Rows	Query
1	346.4 ms	346.4 ms	346.4 ms	0	CREATE INDEX CONCURRENTLY "idx_packages_nuget_metadata_on_package_id_and_normalize_version" ON "packages_nuget_metadata" (package_id, normalized_version)
2	0.0 ms	0.0 ms	0.0 ms	2	SELECT pg_backend_pid()

Histogram for AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion

Query Runtime	Count
0 seconds - 0.01 seconds	0
0.01 seconds - 0.1 seconds	2
0.1 seconds - 1 second	1
1 second - 5 seconds	0
5 seconds - 15 seconds	0
15 seconds - 5 minutes	0
5 minutes +	0

Migration: 20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages

• Type: Regular
• Duration: 56.6 s
• Database size change: +48.27 MiB

Calls	Total Time	Max Time	Mean Time	Rows	Query
1	53542.9 ms	53542.9 ms	53542.9 ms	0	CREATE INDEX CONCURRENTLY "index_packages_packages_on_project_id_and_lower_name" ON "packages_packages" (project_id, LOWER(name)) WHERE package_type = 4
2	0.0 ms	0.0 ms	0.0 ms	2	SELECT pg_backend_pid()

Histogram for AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages

Query Runtime	Count
0 seconds - 0.01 seconds	0
0.01 seconds - 0.1 seconds	2
0.1 seconds - 1 second	0
1 second - 5 seconds	0
5 seconds - 15 seconds	0
15 seconds - 5 minutes	1
5 minutes +	0

Other information

No other migrations pending on GitLab.com

Clone details

Clone ID	Clone Created At	Clone Data Timestamp	Expected Removal Time
database-testing-2153889-10709102-main	2023-07-26T20:54:45Z	2023-07-26T20:11:34Z	2023-07-27 09:01:32 +0000
database-testing-2153889-10709102-ci	2023-07-26T20:54:45Z	2023-07-26T16:46:10Z	2023-07-27 09:01:32 +0000

Job artifacts

Database migrations (on the ci database)

Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

Migration	Type	Total runtime	Result	DB size change
20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum	Regular	3.8 s		+0.00 B
20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion	Regular	3.1 s		+8.00 KiB [note]
20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages	Regular	3.9 s		+8.00 KiB [note]

Runtime Histogram for all migrations

Query Runtime	Count
0 seconds - 0.01 seconds	0
0.01 seconds - 0.1 seconds	11
0.1 seconds - 1 second	0
1 second - 5 seconds	0
5 seconds - 15 seconds	0
15 seconds - 5 minutes	0
5 minutes +	0

Migration: 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum

• Type: Regular
• Duration: 3.8 s
• Database size change: +0.00 B

Calls	Total Time	Max Time	Mean Time	Rows	Query
1	10.3 ms	10.3 ms	10.3 ms	0	ALTER TABLE packages_nuget_metadata ADD CONSTRAINT check_9973c0cc33 CHECK ( char_length(normalized_version) <= 255 ) NOT VALID
1	7.4 ms	7.4 ms	7.4 ms	0	ALTER TABLE "packages_nuget_metadata" ADD "normalized_version" text
1	1.6 ms	1.6 ms	1.6 ms	0	ALTER TABLE packages_nuget_metadata VALIDATE CONSTRAINT check_9973c0cc33
2	0.0 ms	0.0 ms	0.0 ms	2	SELECT pg_backend_pid()

Histogram for AddNormalizedVersionToPackagesNugetMetadatum

Query Runtime	Count
0 seconds - 0.01 seconds	0
0.01 seconds - 0.1 seconds	5
0.1 seconds - 1 second	0
1 second - 5 seconds	0
5 seconds - 15 seconds	0
15 seconds - 5 minutes	0
5 minutes +	0

Migration: 20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion

• Type: Regular
• Duration: 3.1 s
• Database size change: +8.00 KiB ^[note]

Calls	Total Time	Max Time	Mean Time	Rows	Query
1	18.7 ms	18.7 ms	18.7 ms	0	CREATE INDEX CONCURRENTLY "idx_packages_nuget_metadata_on_package_id_and_normalize_version" ON "packages_nuget_metadata" (package_id, normalized_version)
2	0.0 ms	0.0 ms	0.0 ms	2	SELECT pg_backend_pid()

Histogram for AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion

Query Runtime	Count
0 seconds - 0.01 seconds	0
0.01 seconds - 0.1 seconds	3
0.1 seconds - 1 second	0
1 second - 5 seconds	0
5 seconds - 15 seconds	0
15 seconds - 5 minutes	0
5 minutes +	0

Migration: 20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages

• Type: Regular
• Duration: 3.9 s
• Database size change: +8.00 KiB ^[note]

Calls	Total Time	Max Time	Mean Time	Rows	Query
1	42.0 ms	42.0 ms	42.0 ms	0	CREATE INDEX CONCURRENTLY "index_packages_packages_on_project_id_and_lower_name" ON "packages_packages" (project_id, LOWER(name)) WHERE package_type = 4
2	0.0 ms	0.0 ms	0.0 ms	2	SELECT pg_backend_pid()

Histogram for AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages

Query Runtime	Count
0 seconds - 0.01 seconds	0
0.01 seconds - 0.1 seconds	3
0.1 seconds - 1 second	0
1 second - 5 seconds	0
5 seconds - 15 seconds	0
15 seconds - 5 minutes	0
5 minutes +	0

Other information

No other migrations pending on GitLab.com

Clone details

Clone ID	Clone Created At	Clone Data Timestamp	Expected Removal Time
database-testing-2153889-10709102-main	2023-07-26T20:54:45Z	2023-07-26T20:11:34Z	2023-07-27 09:01:32 +0000
database-testing-2153889-10709102-ci	2023-07-26T20:54:45Z	2023-07-26T16:46:10Z	2023-07-27 09:01:32 +0000

Job artifacts

---

Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

added database-testing-automation label

changed the description

Hi @ck3g a database review is needed here, would you mind doing it if you have the capacity? Thank you

Hi @dmeshcharakou would you mind doing the backend review if you have the capacity? Thank you

requested review from @ck3g

requested review from @dmeshcharakou

removed [deprecated] Accepting merge requests label

mentioned in issue #273697 (closed)

added 1 commit

• 8b8b2c60 - Rename normalize function

Compare with previous version

changed the description

added workflowready for review label and removed workflowin dev label

changed the description

requested review from @ghavenga and removed review request for @ck3g

added workflowin review label and removed workflowready for review label

removed review request for @dmeshcharakou

added 1344 commits

• 8b8b2c60...2ac30178 - 1341 commits from branch master
• 53abd3c0 - Merge branch 'master' into 273697-support-nuget-normalized-versions
• 517b0001 - Merge branch 'master' into 273697-support-nuget-normalized-versions
• 6777f021 - Regex match nuget version in normalize_nuget_version function

Compare with previous version

Allure report

allure-report-publisher generated test report!

e2e-test-on-gdk: test report for 027bf023

expand test summary

+------------------------------------------------------------------+
|                          suites summary                          |
+-------------+--------+--------+---------+-------+-------+--------+
|             | passed | failed | skipped | flaky | total | result |
+-------------+--------+--------+---------+-------+-------+--------+
| Data Stores | 20     | 0      | 0       | 8     | 20    | ❗     |
| Package     | 0      | 0      | 1       | 0     | 1     | ➖     |
| Verify      | 8      | 0      | 0       | 2     | 8     | ❗     |
| Manage      | 12     | 0      | 1       | 12    | 13    | ❗     |
| Govern      | 34     | 0      | 0       | 9     | 34    | ❗     |
| Plan        | 47     | 0      | 0       | 6     | 47    | ❗     |
| Create      | 38     | 0      | 0       | 13    | 38    | ❗     |
+-------------+--------+--------+---------+-------+-------+--------+
| Total       | 159    | 0      | 2       | 50    | 161   | ❗     |
+-------------+--------+--------+---------+-------+-------+--------+

e2e-package-and-test: test report for 027bf023

expand test summary

+--------------------------------------------------------------+
|                        suites summary                        |
+---------+--------+--------+---------+-------+-------+--------+
|         | passed | failed | skipped | flaky | total | result |
+---------+--------+--------+---------+-------+-------+--------+
| Package | 70     | 0      | 6       | 0     | 76    | ✅     |
+---------+--------+--------+---------+-------+-------+--------+
| Total   | 70     | 0      | 6       | 0     | 76    | ✅     |
+---------+--------+--------+---------+-------+-------+--------+

requested review from @dmeshcharakou

approved this merge request

requested review from @sabrams and removed review request for @ghavenga

@ghavenga, thanks for approving this merge request.

This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.

For more info, please refer to the following links:

• rspec predictive jobs
• jest predictive jobs
• merging a merge request.

added pipeline:mr-approved label

removed review request for @sabrams

added 828 commits

• 6777f021...bcd13d2d - 826 commits from branch master
• 6bbb58ee - Merge branch 'master' into 273697-support-nuget-normalized-versions
• 5b49c3ba - Check for client_version in PackageFinder

Compare with previous version

added 132 commits

• 5b49c3ba...8bd5f5b7 - 131 commits from branch master
• a684ac3a - Merge branch 'master' into 273697-support-nuget-normalized-versions

Compare with previous version

added 1 commit

• 5fa01dc2 - Fix normalize_nuget_version regex

Compare with previous version

added Data WarehouseNot Impacted label and removed Data WarehouseImpact Check label

requested review from @mayra-cabrera

mentioned in issue #214674 (closed)

mentioned in issue #270976 (closed)

removed review request for @dmeshcharakou