Skip to content
Snippets Groups Projects

Add a setting for allowing/disallowing duplicate NuGet package uploads

What does this MR do and why?

Context

When using the GitLab Package Registry to publish NuGet packages, a duplicate package name/version can be uploaded. This may be great for snapshots, but you may want your releases to be immutable.

This MR introduces a new setting that enables the user to define, at the group level, whether duplicate NuGet packages are allowed or not. In a subsequent MR, the newly added settings in the MR would be utilized in services and endpoints to allow/disallow NuGet packages' duplicate uploads. ALL MRs should be behind the same feature flag.

Implementation

The setting is defined in namespace_package_settings database table:

  • nuget_duplicates_allowed is a boolean column that is true by default (the current default behavior is to allow duplicates). When toggled to false, the duplication should be disallowed.
  • nuget_duplicate_exception_regex is a charvar column to save a regex defining NuGet packages that are allowed to be duplicated when duplicates are not allowed.

In this MR, the setting can be updated using GraghQL, but they are behind the nuget_duplicates_option feature flag. When the feature flag is disabled for the namespace, the setting cannot be updated.

How to set up and validate locally

  1. Try to update the namespace package settings nuget_duplicates_allowed using the query below in graphql-explorer:

    mutation {
      updateNamespacePackageSettings(input: {
        namespacePath: "<your-namespace-full-path>", 
        nugetDuplicatesAllowed:false,
      }) {
        packageSettings {
    	nugetDuplicatesAllowed
        }
      }
    }
  2. The setting shouldn't be updated because the feature flag nuget_duplicates_option is disabled for the namespace.

  3. In the rails console, enable the FF for the namespace:

    Feature.enable(:nuget_duplicates_option, Namespace.find(<namespace_id>))
  4. Restart your GDK and repeat the above graghql query. The setting nuget_duplicates_allowed should be updated to be false

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #293748 (closed)

Edited by Moaz Khalifa

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Moaz Khalifa changed the description

    changed the description

  • Moaz Khalifa requested review from @radbatnag

    requested review from @radbatnag

  • Moaz Khalifa requested review from @carlad-gl

    requested review from @carlad-gl

  • Contributor

    Database migrations (on the main database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20230613182541 - AddNugetDuplicateAllowedToNamespacePackageSettings Regular 4.1 s :white_check_mark: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 8
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20230613182541 - AddNugetDuplicateAllowedToNamespacePackageSettings

    • Type: Regular
    • Duration: 4.1 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 28.2 ms 28.2 ms 28.2 ms 0
    ALTER TABLE "namespace_package_settings" ADD "nuget_duplicates_allowed" boolean DEFAULT TRUE NOT NULL
    1 18.1 ms 18.1 ms 18.1 ms 0
    ALTER TABLE namespace_package_settings VALIDATE CONSTRAINT check_eedcf85c48
    1 12.1 ms 12.1 ms 12.1 ms 0
    ALTER TABLE namespace_package_settings ADD CONSTRAINT check_eedcf85c48 CHECK ( char_length(nuget_duplicate_exception_regex) <= 255 ) NOT VALID
    1 0.8 ms 0.8 ms 0.8 ms 0
    ALTER TABLE "namespace_package_settings" ADD "nuget_duplicate_exception_regex" text DEFAULT '' NOT NULL
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT $1::regtype::oid
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    Histogram for AddNugetDuplicateAllowedToNamespacePackageSettings
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 8
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20230621072848 - AddTextLimitToCiVariableDescription Post deploy 2.2 s :white_check_mark: +0.00 B
    20230621083052 - AddTextLimitToCiGroupVariableDescription Post deploy 2.1 s :white_check_mark: +0.00 B
    20230621102941 - ReplaceOldFkCiJobArtifactsToBuildsV2 Post deploy 1.9 s :white_check_mark: +0.00 B
    20230621103000 - ReplaceOldFkCiRunningBuildsToBuildsV2 Post deploy 1.7 s :white_check_mark: +0.00 B
    20230621103043 - ReplaceOldFkCiJobVariablesToBuildsV2 Post deploy 1.8 s :white_check_mark: +0.00 B
    20230626072436 - DropTmpIndexJobArtifactsIdAndExpireAt Post deploy 2.5 s :white_check_mark: +0.00 B
    20230627174139 - AddIndexToPoolRepositories Post deploy 4.5 s :warning: -3.62 MiB
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-2087657-10526645-main 2023-07-06T06:05:46Z 2023-07-06T04:12:59Z 2023-07-06 18:11:01 +0000
    database-testing-2087657-10526645-ci 2023-07-06T06:05:46Z 2023-07-06T04:48:13Z 2023-07-06 18:11:01 +0000

    Job artifacts

    Database migrations (on the ci database)

    Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).

    Migration Type Total runtime Result DB size change
    20230613182541 - AddNugetDuplicateAllowedToNamespacePackageSettings Regular 4.0 s :white_check_mark: +0.00 B
    Runtime Histogram for all migrations
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 8
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Migration: 20230613182541 - AddNugetDuplicateAllowedToNamespacePackageSettings

    • Type: Regular
    • Duration: 4.0 s
    • Database size change: +0.00 B
    Calls Total Time Max Time Mean Time Rows Query
    1 36.8 ms 36.8 ms 36.8 ms 0
    ALTER TABLE "namespace_package_settings" ADD "nuget_duplicates_allowed" boolean DEFAULT TRUE NOT NULL
    1 8.7 ms 8.7 ms 8.7 ms 0
    ALTER TABLE namespace_package_settings ADD CONSTRAINT check_eedcf85c48 CHECK ( char_length(nuget_duplicate_exception_regex) <= 255 ) NOT VALID
    1 0.6 ms 0.6 ms 0.6 ms 0
    ALTER TABLE "namespace_package_settings" ADD "nuget_duplicate_exception_regex" text DEFAULT '' NOT NULL
    1 0.4 ms 0.4 ms 0.4 ms 0
    ALTER TABLE namespace_package_settings VALIDATE CONSTRAINT check_eedcf85c48
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT pg_backend_pid()
    2 0.0 ms 0.0 ms 0.0 ms 2
    SELECT $1::regtype::oid
    Histogram for AddNugetDuplicateAllowedToNamespacePackageSettings
    Query Runtime Count
    0 seconds - 0.01 seconds 0
    0.01 seconds - 0.1 seconds 8
    0.1 seconds - 1 second 0
    1 second - 5 seconds 0
    5 seconds - 15 seconds 0
    15 seconds - 5 minutes 0
    5 minutes + 0

    Other information

    Other migrations pending on GitLab.com
    Migration Type Total runtime Result DB size change
    20230621072848 - AddTextLimitToCiVariableDescription Post deploy 11.3 s :warning: +0.00 B
    20230621083052 - AddTextLimitToCiGroupVariableDescription Post deploy 4.2 s :warning: +0.00 B
    20230621102941 - ReplaceOldFkCiJobArtifactsToBuildsV2 Post deploy 3.5 s :warning: +0.00 B
    20230621103000 - ReplaceOldFkCiRunningBuildsToBuildsV2 Post deploy 2.5 s :white_check_mark: +0.00 B
    20230621103043 - ReplaceOldFkCiJobVariablesToBuildsV2 Post deploy 2.5 s :white_check_mark: +0.00 B
    20230626072436 - DropTmpIndexJobArtifactsIdAndExpireAt Post deploy 3.4 s :white_check_mark: +0.00 B
    20230627174139 - AddIndexToPoolRepositories Post deploy 3.4 s :white_check_mark: +0.00 B
    Clone details
    Clone ID Clone Created At Clone Data Timestamp Expected Removal Time
    database-testing-2087657-10526645-main 2023-07-06T06:05:46Z 2023-07-06T04:12:59Z 2023-07-06 18:11:01 +0000
    database-testing-2087657-10526645-ci 2023-07-06T06:05:46Z 2023-07-06T04:48:13Z 2023-07-06 18:11:01 +0000

    Job artifacts


    Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic

  • mentioned in issue #293748 (closed)

  • Moaz Khalifa changed the description

    changed the description

  • Moaz Khalifa removed review request for @carlad-gl and @radbatnag

    removed review request for @carlad-gl and @radbatnag

  • 105 104 raise ExtractionError, 'nuspec file not found' unless entry
    106 105 raise ExtractionError, 'nuspec file too big' if MAX_FILE_SIZE < entry.size
    107 106
    108 Tempfile.open("nuget_extraction_package_file_#{@package_file_id}") do |file|
    107 Tempfile.open("nuget_extraction_package_file_#{package_file.id}") do |file|
  • Moaz Khalifa added 1984 commits

    added 1984 commits

    • 7b451a04...710b05dd - 1982 commits from branch master
    • bc49e0eb - Merge branch...
    • 94ac93bc - Merge branch 'master' into 293748-Add-a-setting-for-disallowing-duplicate-NuGet-package-uploads

    Compare with previous version

  • Moaz Khalifa added 1 commit

    added 1 commit

    • 1696efe3 - Merge branch 'master' into 293748-Add-a-setting-for-disallowing-duplicate-NuGet-package-uploads

    Compare with previous version

  • Moaz Khalifa added 1 commit

    added 1 commit

    • 28d5cbd5 - Merge branch 'master' into 293748-Add-a-setting-for-disallowing-duplicate-NuGet-package-uploads

    Compare with previous version

  • Moaz Khalifa added 765 commits

    added 765 commits

    • 28d5cbd5...36f6d856 - 764 commits from branch master
    • 499bc541 - Add a setting for allowing/disallowing duplicate NuGet package uploads

    Compare with previous version

  • Moaz Khalifa added 1 commit

    added 1 commit

    • d63f0cf6 - Add a setting for allowing/disallowing duplicate NuGet package uploads

    Compare with previous version

  • Moaz Khalifa changed the description

    changed the description

  • Moaz Khalifa requested review from @dbiryukov and @radbatnag

    requested review from @dbiryukov and @radbatnag

  • added workflowin review label and removed workflowin dev label

  • Dmytro Biryukov approved this merge request

    approved this merge request

  • :wave: @dbiryukov, thanks for approving this merge request.

    This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.

    For more info, please refer to the following links:

  • Dmytro Biryukov requested review from @dstull and removed review request for @dbiryukov

    requested review from @dstull and removed review request for @dbiryukov

  • Dmytro Biryukov requested review from @ck3g and removed review request for @dstull

    requested review from @ck3g and removed review request for @dstull

  • Dmytro Biryukov requested review from @ahegyi and removed review request for @ck3g

    requested review from @ahegyi and removed review request for @ck3g

  • Adam Hegyi
  • Adam Hegyi approved this merge request

    approved this merge request

  • added databaseapproved label and removed databasereviewed label

  • Moaz Khalifa added 1 commit

    added 1 commit

    • c842302a - Add a setting for allowing/disallowing duplicate NuGet package uploads

    Compare with previous version

    • Resolved by Moaz Khalifa

      @mkhalifa3

      Step 9 is timing out for me. I have the object storage enabled on my GDK. Is there anything else I need to setup for local testing?

      rad@mjolnirv3 ~/p/t/n/hello (main)> nuget push Package.1.0.0.nupkg -Source localhost
      WARNING: You are running the 'push' operation with an 'HTTP' source, 'http://gdk.test:3000/api/v4/projects/7/packages/nuget/index.json'. Non-HTTPS access will be removed in a future version. Consider migrating to an 'HTTPS' source.
      WARNING: No API Key was provided and no API Key could be found for 'http://gdk.test:3000/api/v4/projects/7/packages/nuget'. To save an API Key for a source use the 'setApiKey' command.
      Pushing Package.1.0.0.nupkg to 'http://gdk.test:3000/api/v4/projects/7/packages/nuget'...
      WARNING: You are running the 'push' operation with an 'HTTP' source, 'http://gdk.test:3000/api/v4/projects/7/packages/nuget/'. Non-HTTPS access will be removed in a future version. Consider migrating to an 'HTTPS' source.
        PUT http://gdk.test:3000/api/v4/projects/7/packages/nuget/
      MSBuild auto-detection: using msbuild version '15.0' from '/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/msbuild/15.0/bin'.
      Using credentials from config. UserName: root
      Please provide credentials for: http://gdk.test:3000/api/v4/projects/7/packages/nuget/index.json
      UserName: root
      Password: ************************
      Please provide credentials for: http://gdk.test:3000/api/v4/projects/7/packages/nuget/index.json
      UserName: root
      Password: **************************
      The operation was canceled.
        Unable to read data from the transport connection: Operation aborted.
        Operation aborted
        Pushing took too long. You can change the default timeout of 300 seconds by using the -Timeout <seconds> option with the push command.
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading