Skip to content
Snippets Groups Projects

Support NuGet version normalization

Merged Moaz Khalifa requested to merge 273697-support-nuget-normalized-versions into master

What does this MR do and why?

Problem

Suppose we have a NuGet package named Hello in our GitLab NuGet Repository with version 1.0.7.0. If we tried to pull it using NuGet 3.4+, you would get the following error:

Package 'Hello.1.0.7' is not found on source 'https://example.gitlab.com/api/v4/projects/<your_project_id>/packages/nuget/index.json'.

Notice the version in the error message: it's different than the package's version stored in GitLab's NuGet Repository. The reason is that NuGet 3.4+ normalizes the version number and then sends the normalized string to the NuGet Repository. We try to search for the package using the received normalized version and we don't find the package. Makes sense.

Solution

In this MR, the following solution has been introduced:

  • Create a new column normalized_version in the packages_nuget_metadata table
  • Add Packages::Nuget::NormalizeVersionService to apply normalization rules on the version string.
  • When a new package is created, we save the normalized version in packages_nuget_metadata 's normalized_version column using a before_validation callback in the Packages::Nuget::Metadatum model.
  • Update Packages::Nuget::PackageFinder to search for packages using LOWER(version) or packages_nuget_metadata 's normalized_version.
  • Get rid of ILIKE fuzzing matching in Packages::Nuget::PackageFinder in favor of exact matching to enhance the performance. NuGet client always sends the package's full name so exact matching would suffice.
  • In a subsequent MR, I will introduce a background migration to backfill the normalized_version for the existing NuGet Repository packages.
  • Changes are behind the nuget_normalized_version feature flag.

How to set up and validate locally

  1. Ensure you have the NuGet CLI installed (see nuget docs for links to installation pages).

  2. In a new directory, run nuget spec.

  3. Update the version in the Package.nuspec file to 1.0.7.0.

  4. Run nuget pack in the same directory, you should find a generated file named Package.1.0.7.nupkg.

  5. Notice the NuGet CLI automatically normalized the version to 1.0.7, we need to update the version manually so that we can push the package with the unnormalized version 1.0.7.0:

    1. Run unzip Package.1.0.7.nupkg, and allow the .nuspec file to be replaced.
    2. Edit the .nuspec file and set the version to 1.0.7.0
    3. Compress/zip all of the files in the folder except for Package.1.0.7.nupkg.
    4. Rename the zip archive Package.1.0.7.0.nupkg.
  6. Add a GitLab project as your NuGet source:

    nuget source Add -Name localhost -Source "http://gdk.test:3000/api/v4/projects/<project_id>/packages/nuget/index.json" -UserName <gitlab_username> -Password <personal_access_token>
  7. In rails console, enable the nuget_normalized_version feature flag before publishing the package:

Feature.enable(:nuget_normalized_version, Project.find(<project_id>))
  1. Push the package to your project:

    nuget push Package.1.0.7.0.nupkg.zip -Source localhost
  2. In the UI, navigate to the project, and go to Packages & Registries -> Package Registry. You should see the NuGet package published with version 1.0.7.0.

  3. Create packages.config file in the directory of the published package:

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Package" version="1.0.7.0" />
</packages>
  1. Execute the command: nuget restore -PackagesDirectory . -Source localhost
  2. The package should be pulled successfully.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Database analysis

Related to #273697 (closed) & #270976 (closed)

Edited by Moaz Khalifa

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Moaz Khalifa changed the description

    changed the description

  • Contributor

    Database migrations (on the main database)

    1 Warnings
    :warning: 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum had a query that exceeded
    timing guidelines
    . Run time should not exceed 100ms, but it was 317.39ms. Please consider possible
    options to improve the query performance.
    ALTER TABLE packages_nuget_metadata VALIDATE
    CONSTRAINT check_9973c0cc33

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum Regular 3.6 s :warning: +0.00 B
    20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion Regular 2.6 s :white_check_mark: +8.97 MiB
    20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages Regular 56.6 s :white_check_mark: +48.27 MiB
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 8
    0.1 seconds - 1 second 2
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 1
    5 minutes + 0

    :warning: Migration: 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum

    • Type: Regular
    • Duration: 3.6 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 317.4 ms 317.4 ms 317.4 ms 0
    ALTER TABLE packages_nuget_metadata VALIDATE CONSTRAINT check_9973c0cc33
    1 18.7 ms 18.7 ms 18.7 ms 0
    ALTER TABLE packages_nuget_metadata ADD CONSTRAINT check_9973c0cc33 CHECK ( char_length(normalized_version) <= 255 ) NOT VALID
    1 6.9 ms 6.9 ms 6.9 ms 0
    ALTER TABLE "packages_nuget_metadata" ADD "normalized_version" text
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddNormalizedVersionToPackagesNugetMetadatum
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 4
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion

    • Type: Regular
    • Duration: 2.6 s
    • Database size change: +8.97 MiB
    Calls Total Time Max Time Mean Time Rows Query
    1 346.4 ms 346.4 ms 346.4 ms 0
    CREATE INDEX CONCURRENTLY "idx_packages_nuget_metadata_on_package_id_and_normalize_version" ON "packages_nuget_metadata" (package_id, normalized_version)
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 2
    0.1 seconds - 1 second 1
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages

    • Type: Regular
    • Duration: 56.6 s
    • Database size change: +48.27 MiB
    Calls Total Time Max Time Mean Time Rows Query
    1 53542.9 ms 53542.9 ms 53542.9 ms 0
    CREATE INDEX CONCURRENTLY "index_packages_packages_on_project_id_and_lower_name" ON "packages_packages" (project_id, LOWER(name))
    WHERE package_type = 4
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 2
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 1
    5 minutes + 0

    Other information

    No other migrations pending on GitLab.com

    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-2153889-10709102-main 2023-07-26T20:54:45Z 2023-07-26T20:11:34Z 2023-07-27 09:01:32 +0000
    database-testing-2153889-10709102-ci 2023-07-26T20:54:45Z 2023-07-26T16:46:10Z 2023-07-27 09:01:32 +0000

    Job artifacts

    Database migrations (on the ci database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum Regular 3.8 s :white_check_mark: +0.00 B
    20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion Regular 3.1 s :white_check_mark: +8.00 KiB [note]
    20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages Regular 3.9 s :white_check_mark: +8.00 KiB [note]
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 11
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20230718124213 - AddNormalizedVersionToPackagesNugetMetadatum

    • Type: Regular
    • Duration: 3.8 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 10.3 ms 10.3 ms 10.3 ms 0
    ALTER TABLE packages_nuget_metadata ADD CONSTRAINT check_9973c0cc33 CHECK ( char_length(normalized_version) <= 255 ) NOT VALID
    1 7.4 ms 7.4 ms 7.4 ms 0
    ALTER TABLE "packages_nuget_metadata" ADD "normalized_version" text
    1 1.6 ms 1.6 ms 1.6 ms 0
    ALTER TABLE packages_nuget_metadata VALIDATE CONSTRAINT check_9973c0cc33
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddNormalizedVersionToPackagesNugetMetadatum
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 5
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20230718160522 - AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion

    • Type: Regular
    • Duration: 3.1 s
    • Database size change: +8.00 KiB [note]
    Calls Total Time Max Time Mean Time Rows Query
    1 18.7 ms 18.7 ms 18.7 ms 0
    CREATE INDEX CONCURRENTLY "idx_packages_nuget_metadata_on_package_id_and_normalize_version" ON "packages_nuget_metadata" (package_id, normalized_version)
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddIndexPackagesNugetMetadatumOnPackageIdAndNormalizedVersion
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20230718160749 - AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages

    • Type: Regular
    • Duration: 3.9 s
    • Database size change: +8.00 KiB [note]
    Calls Total Time Max Time Mean Time Rows Query
    1 42.0 ms 42.0 ms 42.0 ms 0
    CREATE INDEX CONCURRENTLY "index_packages_packages_on_project_id_and_lower_name" ON "packages_packages" (project_id, LOWER(name))
    WHERE package_type = 4
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddIndexPackagesPackagesOnProjectIdAndLowerNameToPackages
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 3
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    No other migrations pending on GitLab.com

    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-2153889-10709102-main 2023-07-26T20:54:45Z 2023-07-26T20:11:34Z 2023-07-27 09:01:32 +0000
    database-testing-2153889-10709102-ci 2023-07-26T20:54:45Z 2023-07-26T16:46:10Z 2023-07-27 09:01:32 +0000

    Job artifacts


    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

  • Moaz Khalifa changed the description

    changed the description

  • Moaz Khalifa requested review from @ck3g

    requested review from @ck3g

  • Moaz Khalifa requested review from @dmeshcharakou

    requested review from @dmeshcharakou

  • mentioned in issue #273697 (closed)

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading