Add a setting for allowing/disallowing duplicate NuGet package uploads
What does this MR do and why?
Context
When using the GitLab Package Registry to publish NuGet packages, a duplicate package name/version can be uploaded. This may be great for snapshots, but you may want your releases
to be immutable.
This MR introduces a new setting that enables the user to define, at the group level, whether duplicate NuGet packages are allowed or not. In a subsequent MR, the newly added settings in the MR would be utilized in services and endpoints to allow/disallow NuGet packages' duplicate uploads. ALL MRs should be behind the same feature flag.
Implementation
The setting is defined in namespace_package_settings
database table:
-
nuget_duplicates_allowed
is aboolean
column that istrue
by default (the current default behavior is to allow duplicates). When toggled tofalse
, the duplication should be disallowed. -
nuget_duplicate_exception_regex
is acharvar
column to save a regex defining NuGet packages that are allowed to be duplicated when duplicates are not allowed.
In this MR, the setting can be updated using GraghQL, but they are behind the nuget_duplicates_option
feature flag. When the feature flag is disabled for the namespace, the setting cannot be updated.
How to set up and validate locally
-
Try to update the namespace package settings
nuget_duplicates_allowed
using the query below in graphql-explorer:mutation { updateNamespacePackageSettings(input: { namespacePath: "<your-namespace-full-path>", nugetDuplicatesAllowed:false, }) { packageSettings { nugetDuplicatesAllowed } } }
-
The setting shouldn't be updated because the feature flag
nuget_duplicates_option
is disabled for the namespace. -
In the rails console, enable the FF for the namespace:
Feature.enable(:nuget_duplicates_option, Namespace.find(<namespace_id>))
-
Restart your GDK and repeat the above graghql query. The setting
nuget_duplicates_allowed
should be updated to befalse
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #293748 (closed)
Merge request reports
Activity
changed milestone to %16.2
assigned to @mkhalifa3
added 425 commits
-
dc90d4fb...5aff3896 - 424 commits from branch
master
- 7b451a04 - Merge branch 'master' into 293748-Add-a-setting-for-disallowing-duplicate-NuGet-package-uploads
-
dc90d4fb...5aff3896 - 424 commits from branch
- A deleted user
added database databasereview pending documentation labels
1 Warning c0d5d840: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. 1 Message This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
-
doc/api/graphql/reference/index.md
(Link to current live version) -
doc/development/packages/settings.md
(Link to current live version)
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.md
files that you've changed. The first few lines of each*.md
file identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend Mo Khan (
@mokhax
) (UTC-6)Luke Duncalfe (
@.luke
) (UTC+12)database João Pereira (
@jdrpereira
) (UTC+1)Leonardo da Rosa (
@l.rosa
) (UTC-3)~"migration" No reviewer available No maintainer available To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger-
- A deleted user
added Data WarehouseImpact Check label
added featureaddition label
Allure report
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for c0d5d840expand test summary
+------------------------------------------------------------------+ | suites summary | +-------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +-------------+--------+--------+---------+-------+-------+--------+ | Package | 0 | 0 | 1 | 0 | 1 | ➖ | | Govern | 19 | 0 | 0 | 12 | 19 | ❗ | | Plan | 47 | 0 | 0 | 11 | 47 | ❗ | | Data Stores | 20 | 0 | 0 | 10 | 20 | ❗ | | Verify | 8 | 0 | 0 | 5 | 8 | ❗ | | Create | 19 | 0 | 0 | 17 | 19 | ❗ | | Manage | 12 | 0 | 1 | 12 | 13 | ❗ | +-------------+--------+--------+---------+-------+-------+--------+ | Total | 125 | 0 | 2 | 67 | 127 | ❗ | +-------------+--------+--------+---------+-------+-------+--------+
e2e-package-and-test:
test report for c0d5d840expand test summary
+--------------------------------------------------------------+ | suites summary | +---------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +---------+--------+--------+---------+-------+-------+--------+ | Package | 70 | 0 | 6 | 0 | 76 | ✅ | +---------+--------+--------+---------+-------+-------+--------+ | Total | 70 | 0 | 6 | 0 | 76 | ✅ | +---------+--------+--------+---------+-------+-------+--------+
- Resolved by David Fernandez
- Resolved by David Fernandez
Hi @radbatnag
Can you plz take over the backend review for this MR if you have the capacity?
Hi @carlad-gl
Can you plz take over the database review for this MR if you have the capacity? I'm adding two new columns to the
namespace_package_settings
table.Thank you both
Edited by Moaz Khalifa
requested review from @radbatnag
requested review from @carlad-gl
Database migrations (on the main database)
Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20230613182541 - AddNugetDuplicateAllowedToNamespacePackageSettings Regular 4.1 s +0.00 B Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 8 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230613182541 - AddNugetDuplicateAllowedToNamespacePackageSettings
- Type: Regular
- Duration: 4.1 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 28.2 ms 28.2 ms 28.2 ms 0 ALTER TABLE "namespace_package_settings" ADD "nuget_duplicates_allowed" boolean DEFAULT TRUE NOT NULL
1 18.1 ms 18.1 ms 18.1 ms 0 ALTER TABLE namespace_package_settings VALIDATE CONSTRAINT check_eedcf85c48
1 12.1 ms 12.1 ms 12.1 ms 0 ALTER TABLE namespace_package_settings ADD CONSTRAINT check_eedcf85c48 CHECK ( char_length(nuget_duplicate_exception_regex) <= 255 ) NOT VALID
1 0.8 ms 0.8 ms 0.8 ms 0 ALTER TABLE "namespace_package_settings" ADD "nuget_duplicate_exception_regex" text DEFAULT '' NOT NULL
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT $1::regtype::oid
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
Histogram for AddNugetDuplicateAllowedToNamespacePackageSettings
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 8 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Other information
Other migrations pending on GitLab.com
Migration Type Total runtime Result DB size change 20230621072848 - AddTextLimitToCiVariableDescription Post deploy 2.2 s +0.00 B 20230621083052 - AddTextLimitToCiGroupVariableDescription Post deploy 2.1 s +0.00 B 20230621102941 - ReplaceOldFkCiJobArtifactsToBuildsV2 Post deploy 1.9 s +0.00 B 20230621103000 - ReplaceOldFkCiRunningBuildsToBuildsV2 Post deploy 1.7 s +0.00 B 20230621103043 - ReplaceOldFkCiJobVariablesToBuildsV2 Post deploy 1.8 s +0.00 B 20230626072436 - DropTmpIndexJobArtifactsIdAndExpireAt Post deploy 2.5 s +0.00 B 20230627174139 - AddIndexToPoolRepositories Post deploy 4.5 s -3.62 MiB Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-2087657-10526645-main
2023-07-06T06:05:46Z 2023-07-06T04:12:59Z 2023-07-06 18:11:01 +0000 database-testing-2087657-10526645-ci
2023-07-06T06:05:46Z 2023-07-06T04:48:13Z 2023-07-06 18:11:01 +0000 Database migrations (on the ci database)
Migrations included in this change have been executed on gitlab.com data for testing purposes. For details, please see the migration testing pipeline (limited access).
Migration Type Total runtime Result DB size change 20230613182541 - AddNugetDuplicateAllowedToNamespacePackageSettings Regular 4.0 s +0.00 B Runtime Histogram for all migrations
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 8 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Migration: 20230613182541 - AddNugetDuplicateAllowedToNamespacePackageSettings
- Type: Regular
- Duration: 4.0 s
- Database size change: +0.00 B
Calls Total Time Max Time Mean Time Rows Query 1 36.8 ms 36.8 ms 36.8 ms 0 ALTER TABLE "namespace_package_settings" ADD "nuget_duplicates_allowed" boolean DEFAULT TRUE NOT NULL
1 8.7 ms 8.7 ms 8.7 ms 0 ALTER TABLE namespace_package_settings ADD CONSTRAINT check_eedcf85c48 CHECK ( char_length(nuget_duplicate_exception_regex) <= 255 ) NOT VALID
1 0.6 ms 0.6 ms 0.6 ms 0 ALTER TABLE "namespace_package_settings" ADD "nuget_duplicate_exception_regex" text DEFAULT '' NOT NULL
1 0.4 ms 0.4 ms 0.4 ms 0 ALTER TABLE namespace_package_settings VALIDATE CONSTRAINT check_eedcf85c48
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT pg_backend_pid()
2 0.0 ms 0.0 ms 0.0 ms 2 SELECT $1::regtype::oid
Histogram for AddNugetDuplicateAllowedToNamespacePackageSettings
Query Runtime Count 0 seconds - 0.01 seconds 0 0.01 seconds - 0.1 seconds 8 0.1 seconds - 1 second 0 1 second - 5 seconds 0 5 seconds - 15 seconds 0 15 seconds - 5 minutes 0 5 minutes + 0 Other information
Other migrations pending on GitLab.com
Migration Type Total runtime Result DB size change 20230621072848 - AddTextLimitToCiVariableDescription Post deploy 11.3 s +0.00 B 20230621083052 - AddTextLimitToCiGroupVariableDescription Post deploy 4.2 s +0.00 B 20230621102941 - ReplaceOldFkCiJobArtifactsToBuildsV2 Post deploy 3.5 s +0.00 B 20230621103000 - ReplaceOldFkCiRunningBuildsToBuildsV2 Post deploy 2.5 s +0.00 B 20230621103043 - ReplaceOldFkCiJobVariablesToBuildsV2 Post deploy 2.5 s +0.00 B 20230626072436 - DropTmpIndexJobArtifactsIdAndExpireAt Post deploy 3.4 s +0.00 B 20230627174139 - AddIndexToPoolRepositories Post deploy 3.4 s +0.00 B Clone details
Clone ID Clone Created At Clone Data Timestamp Expected Removal Time database-testing-2087657-10526645-main
2023-07-06T06:05:46Z 2023-07-06T04:12:59Z 2023-07-06 18:11:01 +0000 database-testing-2087657-10526645-ci
2023-07-06T06:05:46Z 2023-07-06T04:48:13Z 2023-07-06 18:11:01 +0000
Brought to you by gitlab-org/database-team/gitlab-com-database-testing. Epic
mentioned in issue #293748 (closed)
- A deleted user
added database-testing-automation label
- Resolved by David Fernandez
Just curious since the UI settings won't be available in this MR (?): What is the default behavior? Allow or disallow duplicates?
removed review request for @carlad-gl and @radbatnag
mentioned in merge request gitlab-com/www-gitlab-com!126510 (merged)
105 104 raise ExtractionError, 'nuspec file not found' unless entry 106 105 raise ExtractionError, 'nuspec file too big' if MAX_FILE_SIZE < entry.size 107 106 108 Tempfile.open("nuget_extraction_package_file_#{@package_file_id}") do |file| 107 Tempfile.open("nuget_extraction_package_file_#{package_file.id}") do |file| changed this line in version 3 of the diff
added 1984 commits
-
7b451a04...710b05dd - 1982 commits from branch
master
- bc49e0eb - Merge branch...
- 94ac93bc - Merge branch 'master' into 293748-Add-a-setting-for-disallowing-duplicate-NuGet-package-uploads
-
7b451a04...710b05dd - 1982 commits from branch
added 1 commit
- 1696efe3 - Merge branch 'master' into 293748-Add-a-setting-for-disallowing-duplicate-NuGet-package-uploads
added 1 commit
- 28d5cbd5 - Merge branch 'master' into 293748-Add-a-setting-for-disallowing-duplicate-NuGet-package-uploads
added 765 commits
-
28d5cbd5...36f6d856 - 764 commits from branch
master
- 499bc541 - Add a setting for allowing/disallowing duplicate NuGet package uploads
-
28d5cbd5...36f6d856 - 764 commits from branch
added 1 commit
- d63f0cf6 - Add a setting for allowing/disallowing duplicate NuGet package uploads
requested review from @dbiryukov and @radbatnag
added workflowin review label and removed workflowin dev label
- Resolved by Adam Hegyi
added databasereviewed label and removed databasereview pending label
@dbiryukov
, thanks for approving this merge request.This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.
For more info, please refer to the following links:
added pipeline:mr-approved label
requested review from @dstull and removed review request for @dbiryukov
- Resolved by David Fernandez
added databaseapproved label and removed databasereviewed label
- Resolved by David Fernandez
- Resolved by David Fernandez
- Resolved by Ahmed Hemdan
added Data WarehouseNot Impacted label and removed Data WarehouseImpact Check label
added 1 commit
- c842302a - Add a setting for allowing/disallowing duplicate NuGet package uploads
- Resolved by Moaz Khalifa
Step 9 is timing out for me. I have the object storage enabled on my GDK. Is there anything else I need to setup for local testing?
rad@mjolnirv3 ~/p/t/n/hello (main)> nuget push Package.1.0.0.nupkg -Source localhost WARNING: You are running the 'push' operation with an 'HTTP' source, 'http://gdk.test:3000/api/v4/projects/7/packages/nuget/index.json'. Non-HTTPS access will be removed in a future version. Consider migrating to an 'HTTPS' source. WARNING: No API Key was provided and no API Key could be found for 'http://gdk.test:3000/api/v4/projects/7/packages/nuget'. To save an API Key for a source use the 'setApiKey' command. Pushing Package.1.0.0.nupkg to 'http://gdk.test:3000/api/v4/projects/7/packages/nuget'... WARNING: You are running the 'push' operation with an 'HTTP' source, 'http://gdk.test:3000/api/v4/projects/7/packages/nuget/'. Non-HTTPS access will be removed in a future version. Consider migrating to an 'HTTPS' source. PUT http://gdk.test:3000/api/v4/projects/7/packages/nuget/ MSBuild auto-detection: using msbuild version '15.0' from '/Library/Frameworks/Mono.framework/Versions/6.12.0/lib/mono/msbuild/15.0/bin'. Using credentials from config. UserName: root Please provide credentials for: http://gdk.test:3000/api/v4/projects/7/packages/nuget/index.json UserName: root Password: ************************ Please provide credentials for: http://gdk.test:3000/api/v4/projects/7/packages/nuget/index.json UserName: root Password: ************************** The operation was canceled. Unable to read data from the transport connection: Operation aborted. Operation aborted Pushing took too long. You can change the default timeout of 300 seconds by using the -Timeout <seconds> option with the push command.