Skip to content

Fix nuget uploads

David Fernandez requested to merge 10io-fix-nuget-upload into master

What does this MR do?

This MR is a partial extraction from !21561 (merged).

This MR is part of the nuget MVC: #20050 (closed). As such, this MVC is behind a feature flag. No documentation or changelog has been added.

This MR does the following:

  • Previously when uploading a nuget package a fixed package name and a fixed version were set. Those are updated by the worker described above. The issue is that if we have a second upload, it will use the same fixed package name and version and this will lead to a database insert failure: there is a unique index on project_id, package_name, package_version. To fix that, the package version has a fixed prefix and a random uuid.
  • Nuget uploads were not properly handled by workhorse. This has been fixed (gitlab-workhorse!451 (merged)) and the uploaded file is now received by rails under the package param name (and not file as previously). Also this upload is a multipart upload, as such it lacks of the length header.

Screenshots

Uploading a package with nuget:

$ nuget push DummyProject.DummyPackage.1.0.0.nupkg  -source "locally"
WARNING: No API Key was provided and no API Key could be found for 'https://gitlab.local:3443/api/v4/projects/1/packages/nuget'. To save an API Key for a source use the 'setApiKey' command.
Pushing DummyProject.DummyPackage.1.0.0.nupkg to 'https://gitlab.local:3443/api/v4/projects/1/packages/nuget'...
  PUT https://gitlab.local:3443/api/v4/projects/1/packages/nuget/
  Created https://gitlab.local:3443/api/v4/projects/1/packages/nuget/ 32050ms
Your package was pushed.

Inspecting the package file on the rails backend:

[5] pry(main)> Packages::Package.nuget.last.package_files.first.file.size
  Packages::Package Load (0.5ms)  SELECT  "packages_packages".* FROM "packages_packages" WHERE "packages_packages"."package_type" = 4 ORDER BY "packages_packages"."id" DESC LIMIT 1
  Packages::PackageFile Load (0.5ms)  SELECT  "packages_package_files".* FROM "packages_package_files" WHERE "packages_package_files"."package_id" = 1 ORDER BY "packages_package_files"."id" ASC LIMIT 1
=> 3513
[6] pry(main)> Packages::Package.nuget.last.package_files.first.file.read
  Packages::Package Load (0.6ms)  SELECT  "packages_packages".* FROM "packages_packages" WHERE "packages_packages"."package_type" = 4 ORDER BY "packages_packages"."id" DESC LIMIT 1
  Packages::PackageFile Load (0.4ms)  SELECT  "packages_package_files".* FROM "packages_package_files" WHERE "packages_package_files"."package_id" = 1 ORDER BY "packages_package_files"."id" ASC LIMIT 1
=> "PK\x03\x04\x14\x00\x00\x00\b\x00\eq\x84Ow\xEE\xB7/ \x01\x00\x00\x03\x02\x00\x00\v\x00\x00\x00_rels/.rels\x94\x91\xDDj\xC2@\x10F\xEF}\x8Ae\xEF\xCDn\x8CI6%Q\xD2\xAA\xD0\xBB\"\xBE\xC0t3\x89\xDB\xBA?\xEC\xAER\xDF\xBE\xA1ZPJ\xA1\xBD\x1Cf\xCE\xE1\xFB\x98z\xF9\xA1\x0F\xE4\x84>(k\x1A\x9A&\x9C\x124\xD2v\xCA\f\r=\xC6~*\xE8r1\xA9\xB7x\x808\x9E\x84\xBDr\x81\x8C\x8C\t\r\xDD\xC7\xE8\x1E\x18\vr\x8F\x1ABb\x1D\x9Aq\xD3[\xAF!\x8E\xA3\x1F\x98\x03\xF9\x0E\x03\xB2\x19\xE7\x05\xF3\xB7\x0E\xBA\x98\x10r\xA7%\xBB\xB3\xC3\x1FR\xAD\xA4\xB7\xC1\xF61\x91V_}c\xB6\xD1\x98r\xC6K\xA6\xC1\xA8\x1EC\xA4d\a~\xC0\xD8P\xB6:j}~\xF1\xF6\reL.\xC3%Fb\x8E\xC1\xA1\xA4\xE4\xB9k\xE8v\x95\x8A\xB2\xDDd\xEBM\x9B\xB5\xABYQR\xC2\xFE\x98\xE9_E\x99\xC6\b\x1DD`\xD2z\x9C:?\xD2>*\f7\x89\xBF\xE9\x80\xFE\xA4$\xFE\xCE0|\x15\x00\x12\xB1\x041\x9F\x17<\xAD\xAAR\x00/\x85\xC8A\x16Y\xD5%.\xE8N\xBAk\xC5\xF6\x91WE!\xF2\xA76\x13\xEB<\x17_\x15kv\xF7\xCC\xC5'\x00\x00\x00\xFF\xFF\x03\x00PK\x03\x04\x14\x00\x00\x00\b\x00\eq\x84O\xC6\xF5\xA9\xBD`\x01\x00\x00h\x02\x00\x00 \x00\x00\x00DummyProject.DummyPackage.nuspect\x92KN\xC30\x14E\xE7]\x85\xE51\xC4\xFD\b\t!'U\xF9\r\x10\xAA\x18t\x03\x96\xFD\x9A\x98\xC6\x1F\xEC\x17\xDA\xB25\x06,\x89-\xE0\xD6I\xA9\x90\x902\xB9\xBE\xF7=\x9F\xE4\xE6\xFB\xF3\x8B\xCFw\xA6%\xEF\x10\xA2v\xB6\xA4\x93bL\tX\xE9\x94\xB6uI;\\_^\xD3y5\xE2^\xC8\x8D\xA8\x81\xA4\xB4\x8D%m\x10\xFD\rcQ6`D,\x8C\x96\xC1E\xB7\xC6B:\xC3r6-`\xD3\xF1d\xC6\xC6W\xCCv\xD1\x83,vQ\xD1jD\b7\x80B\t\x14\a\x91\xA4V\xD5}g\xCC\xFE%\xB8W\x90Xd\x91o\xE4,\xB99\xD6SV\t\xB2\x18s6\xC8l\xA2\xC6\x16\xF2\x1A\xE2\x87\xD1|\x98\x03\xA2\xC3\xC6\x85X\xAD \"g\x83\xCA\x9E\xDBZ8Y\xBD\xC8N\x80\xB7N\ax\xD6\x12l\x84\x85\x94\xE0QX\t\xD5Z\xB41\xDD\xF0\xAF\x9F\xC7\x15D\x19\xB4\xC7\x03\xE7\xAA\xD1\x91\xA4G\x10\x95)\xF3\xDBrv\x1E\x1A\xC6<X\x95z\xD0\xD0\x83\xA4\xC3:\xB8\xCE\x13\x14\xA1\x06|\f\xC2\xC0\xD6\x85MI\x8B\xE5\xC3\xEA\xCE\x05Xx?K\xED\r\xF1\xF3-{\xA2UI\x97\xB0Eg\x8F-=Eg\xE9Y\xEB\xD3\xF4Eg\xA9\xF8\x9Dl;\x05%\xBD\xEDt\xAB.\x16V\xB4\xFB\x8F\x14\xA2\x84\x9D(\xD8\x11\xA3\xE7d\x7FA9\xFB\xAD\x96\xF7?\x02T?\x00\x00\x00\xFF\xFF\x03\x00PK\x03\x04\x14\x00\x00\x00\b\x00Yo\x82O^G\x19\xEB\x02\x06\x00\x00\x00\x10\x00\x00\"\x00\x00\x00lib/netcoreapp3.0/DummyProject.dll\xEDV]l\x14U\x14>\xB7\xBB-mik\e(\xD6_\x06\n\xB5\"\xCEnY \x82?\xF4g\xDBR\xE8\x9F\xDDmQ\xC4\x94\xD9\xDD\xDB\xED\xC0\xEC\xCC83[\xD8\xC4\b\x89\xD1D\xC2\x83\t\x98h\x8C\xE1\xC9\xA8\x89F\x1E0A\x84\a\x13\x13\xA2\to\x92\x98\xF8\xC2\x93\xE1\x01\xA3!\xF1Ac\xC4\xEF\xDE\x99\xDD\xEEv\x8B\xE2\x13>pv\xEF\x99{~\xEE\xB9\xDF9\xF7\xCE\xDC;\xB6\xFFm\n\x11Q\x18\xED\xD6-\xA2\xF3\xE4S/\xFD;\x1DGkY{\xA1\x85\xCE5\\Yw\x9E\x8D^Y\x97\x9C\xD7]\xC5v\xAC\xAC\xA3\xE5\x94\xB4f\x9A\x96\xA7\xA4\xB8\xE2\xE4ME7\x95\xF8DB\xC9Y\x19\xAE677n\bbL\x0E\x12\x8D\xB2\x10\x15>Y{\xA3\x18\xF7\x1A\xADWV\xB2(Q=\x84:_\xF7S\x17\x98R\x02\xD6*\xFB5>n\xA2\xC5\xA7\x04U\xE3wC\xD4\xFB\x86p\x15\xFF\xC5g\xE9!\xE9\x14\xE2N\x04\t\xBF\x1AZ&\xC9\x83DMx\x9C\xDCH\x94\xBC\x83\x9A\x94H\xF1\xE1\x17\xA9\x1E\xF2\xEE2Y\xF5\xF8Q\x0F\xCF\euA^\xF5\x8B\xB8\xCBB\x1CT\x1D\xD7IS\x80\r\x18e\xA2\x8D\x95~P\xF7\xAA\x0E7\xAC\xB4\x8FU`\x96\xB1Z\xAA\xFC\xFA\x97\xC2\xBC\xDC\xE5?w\xCB!\xB5\x94\xC1\xA4\xDD\xB5D\xECNr\\\x86\xBA(\xD4\xDD$\x10nZ_\xD3\xDD,;\xD4\x9F\xD8\xD3\xCF\x82\x88\x02\xDF\xC2V5\xAA\xC6\xA2\xB1\x9E\x1DBSK\x06\xF8\xC70w\xBEF\xB4\x1A\xB8;\xD0:\x13\x9E\xA3\x9BYWxl\x0E\xFBiwN'(\x1A\xF6\xD7\xAFsxz$.r\x82|]\xC8\xFD\x86\x95\n0`8\en'j\x10\xC2\x1F,Fk\xFC|Z\x82\xB2\x88&d\xC0\xA3\x95A\xDF/\xFD\xEB\xCCGYG\xBF\xD1\x15VG\xC3L\xF0-t\x81\xDDG?3\xA1\x7F\x9984*{\x1F\xFCK\x12\xFC\a\xC9\x17$?%y\xBB\xB4\x0E\xC9\xFE\x9F\xEC4k\xA46:-\x03\xFB\xD1\x99\xFC\xB5\xD2g,T\xB3M\xF6'\x15\xA1\x7F\x93\x8E\xB1VH\xFB}\xA9\xE3\x1C\xC6\xFBX[k\x1A\xE8\x1C\xB0\xB5\x91\xD0=\x00\xDEH\x8F\x83\xB7R\x8F\xE4;$\xEF\x93|D\xF2\xE7%\x7FQrM\xF2W\xA8\x9B\xDAI\x97\x11T\xE4\xDD\t\xBE\x8A6\x83?D{\xC1;\xB1\xBFUz\x82N\x80\xC7\xE84\xF8\xD3t\x06|\x80.\x82\xEF\xA5o\xC1\x13\xF4=\xF8K\xE0\xE1\xE3K\xF7HH\xBE\x8D5%\xED\x83\xC0)(L\xACLW\xDA\xBA\xCF\x8CY\x99\xBC\xC1\x9F\xA3D\xC1\xF5xN\x1D\xB0L\xD72xQ\x9C\xCA\x9B\x9E\x9E\xE3\xB4\xCF\xD1=>\xAA\x9B\x9C\xE2<\x95\xCFf\xB5\x94\xC1\xFB<\xEC\x8DT\xDE\xE3\xD4\xE7\xBA<\x972\nI\xDD+W'5'\xCB\xBD!|\x84\xF8\x11\xCB9\\\xED?\xA4\e|\x86;\xAEn\x99\xD5\xC6\x11s\xCErr\x9A\a\xA3f\xDC\xD6\vx\xE7\xF4l\xDE\x91~\xD5\xE68w\xD3\x8EnW\x1A\a\xAC\x9C\xAD\er\xC4\x147\xB4\xA3\xB2\xE7V\x0F\x9EtP\x9B\xB4\xB7\xDC\xA49[3\v\x8B\x86\xA0LR\xEF\xE9)\xDD\xD0\xBD2ke-\xD5 \x17\xBCV4je)\x9E\xCF\xE5\xC4T\x87x\xDAS3\x86\x11\xB8\x97F\xF19\x03\x16\f\x10\xDEY\xEE\x90\x9A\xF6,\xA7h\x8E\xEBZ\xD6\xB4\\OO\xBBK\xE7\xF1\xD3\xE4N\x82;\vz\x9A\xBB\xC1\xD2a^\xAC9\xC4\x89\x94\x98\xB2b~\n>\x87\x92\xEC\xD8\x81\xD6\x0F\xBF.\xEC\xF9\xBCa\xEA\x9D\xDF\xBF\xD9\xD4Na\x85\xB1\xFA\x90B\xAC\x16\x9D\xB66!\xB6`[\xB1\x96\xFA\xB3\xBBf\x8F\xB5]m\xDCY\xCF\x82\xAF\xED\xA3b\xB3%k\xD6\xECs4{\xDC2\a\x8F\xA6\xB9\\\x84\xE4\xBCc\x1Dq\x19\xFCV\xC8\xCD\xB8\x8DQ\x87:>\x98\x1C\xB0\x1C\xDEg\xDB\x9B\x83\xDA<\xBB\x10S\xA3\b\xD0\xB2\xBA\xB4{\xE2\xBAk\eZa\x1C\"50\n'\xB9\v\xA4\x8D\x8CjeZD\x0F3\xBA_\x1E|\xF8kJFd%\xCE@\x99\x165a\xC2\x1E|\xEE\xF0\xF3\a\xF5\xF8\xDD6FM\x15\x05\b\xE8\xBB\xD8\xE0\x19bc\x93\xE2\xBD?\x89v\r\xE7\xCE\xB5\x15\x8Bo\x99\x00\xBF\n\xED14\x03\xDFn\xA3\xFC\x90\xA1\xCA\xB3M\xD0T\"\x9EH\xAC\xEB\v\xFF\xF8\xE9\xA5\xE1\x8B\xD3\x87\x1Fy\xEF\xD7\x0FR\"Fd\xDAE\xC6\x91\x8C\xB6\xA0g\"\x01Z7\x82#D3f\r-\x97r#f\x1E\xEF\xD0\xEC\xBCff\xDCY\xCB\x8C\x94\x83\xAD\x14\xAC\xD4\xA1\x88,E\xC4\xE4^\x1A\xF5\xD4l\eU\xACpR\xEDL\x8A\x12\xBB\xFB\xB6l\xDBN\x01\x9C\x8F\x8ApD\xCA_\xBC\xD5\xDC\x1A\xBE\xC9\x9E|wt\xF2\xE6W]\x8B\xF0\xAF\x16\x8F\xFDe\xE8rW\xB94\x8B\x95\x8C\e\xC6\x98\xA6\x9B\x94s\x05\f.7\xB5\xA4[\e\x11dii\xEEQ\x151Y\xA4\x0E\xFF\x16U\xA1\x17{&\xBA\x8C^\x90\xB8;\xBC\xD0\x8Bu*\xBB?)\xA1\xAD\xE0381f\xC1\ai\n\xBD\x11\xDC\xB4\xC6!\x8F\x80\x0F\xF9\xB7.\xBA\x14\xFE\xE5/?\x0E\xAB\x88\xB9+\x90\xC4\xB9\xBF\xE4ZDq\xE95\x83s\xCDA\x1C\x1D\xF7\a\x8E\x98&\xCD\x91%\xED\e\xE4\xA8$\xAC\x1A\xB4.\xEC\x1Ay\xF0\xB3 \xF9t6|\\\\\x12\x80\xC9\x83\x97\x0E}v\x99H\a\xA4O\xB4\xF4\xDBJ)Q\x03\xE4\xDB\x01\xFD\x00|r\xF8q\xF8{\xE4\xCA\x19\xE7\x11\xC3\xC5V\xF3\xB9\x86\x96\xA1\xBC\xF4*\xA0oc.\x8B\x0EaD\x9A\xC4\v\xBF\tw\x9Fb\x1C[b-\xA02\x9A\x8CI2\x1EG\x1C\xFF\xD3\xD0\x8F\xDB\n+a\x8CKKZb\xB7+r\x8B\x97\xE6\x9B\\2\x9BX\xBF\xFA\xB2\x183h\x0E\xA2,\x8E\xED\xC1\xD9\x1E-51g\e\xFCGd~\xC2\xD7\x046\xA3\f\xE1\xED\xE6R\x91\xB5A\xFE\xAB\xD7-o\x04\xA3\xB0d\xE5h\x91\xAD\x8D\x11\x02y\x16\xF5\x12\xC8\x14\xB4=r\xAE\x89@\xAF\as\x15\xB1\x9A\xFFi\xCE\x9D\xB2V\xBE]\xD4_\xD8\xCB\xEB\xFA\xCF5\xAA\xAB\x1A\xBB\xB4R=e5zJ\xD6\xB4\x0FV\x17^9\xEC\x10C\xAE\xF5\xED\xC7\xF8\xE3\xFE7\xA4\xF8w\xF2\xEB\xDB\xEF6\x90{t7\xE8oPK\x03\x04\x14\x00\x00\x00\b\x00\eq\x84OA\xDF\xB6Q\xD3\x00\x00\x00\xCB\x01\x00\x00\x13\x00\x00\x00[Content_Types].xml\x9C\x91=n\xC30\f\x85\xF7\x9CB\xE0Z\xD8r3\x04Ea;C\xDA\e\xE4\x02\x82L\xDBBe\x8A\x10\xE9 \xBD}\x95\x9F\xA9@\xD3\xA2#\xF1\xBD\xF7\xBE\x81\xED\xFE\xBCDs\xC2,!Q\a\xCFu\x03\x06\xC9\xA7!\xD0\xD4\xC1\xAAc\xF5\x02\xFB~\xD3\x1E?\x19\xC5\x94,I\a\xB3*\xBFZ+~\xC6\xC5I\x9D\x18\xA9\x901\xE5\xC5i9\xF3d\xD9\xF9\x0F7\xA1\xDD6\xCD\xCE\xFAD\x8A\xA4\x95^6\xA0\xDF\x18\xD3\xBE\xE1\xE8\xD6\xA8\xE6\xFD\\\xC8M\x9D1\n\x98\xC3-{\xD1u\xE0\x98c\xF0N\v\xB7'\x1A\xBE\x89\xAA\xBB\xA4.\xCDkF\xE6\xC0\xF2T\x02`\xECO\x16\x96e\xF0\xFC?\x8FO\x19+\xCE\x85f\r\xF8\x9Bi\x88\xF1\x81&yE}\xD0\xA6U\x18\xFD\x9F\x06Z{}N\xFF\x05\x00\x00\xFF\xFF\x03\x00PK\x03\x04\x14\x00\x00\x00\b\x00\eq\x84O\x1A\xDBV\x92p\x01\x00\x00l\x02\x00\x00Q\x00\x00\x00package/services/metadata/core-properties/eb8aacee7a8446019978a07885ac639d.psmdcplRKo\xE3 \x10\xBE\xE7W \x9F\x13\x88\xD7\x9B\xA8\xCA\xDAT\xEAV\xEAa\xB7\xABH\eU\xBDR\x98$ll\xB0\x98\xA1n\xFE\xFDb\xC7\xB5\xD2\xAA\x12\a\x98\xEF1/\xCA\xDB\xB7\xA6f\xAF\x10\xD0zWe9_f\f\x9C\xF6\xC6\xBAC\x95E\xDA/n\xB2[9+\xB5\x0F\xB0\r\xBE\x85@\x16\x90%\x91\xC3\x8D\xD1Uv$j7B\xB41\xD4\xDC\x87\x830Z@\r\r8B\x91\xF3\\d\x13\x97 4\xF8\xA5`@&\xE6\e\xDA\x89\xD5u\x1D\xEF\x8A\x81\xF7m\xB9\xCC\xC5\xF3\xE3\xEF\xBF\xFA\b\x8DZX\x87\xA4\x9C\x86Q5)p\x80\x91\xA7R]B\xF6>4\x8Apph\x95>\xA9\x03\xF4Nk\xD1\x00)\xA3H\x89\xBE\xB3E;\xB5\x96\xC9\x19c\xA5\xD1\e\x1D@\x91\x0Fr\aH\xA5\xB8\n\x8C\xB8\x01\xD4\xC1\xB6\x94\xE6&wG\x8B,\x1D\xC5Ll\x9A3Kv\xFF@_d\xD7\xBCQjM\x1A\x8F\xDD[\b\xF2\xBE\xE7o/t~y\\\xAA\x1C\xB4W\xC4^:\xAEI\xA6-\xF1e)\xDE\x9F=t\x82s\xE7\x83AY\x8A\xE9\xDA\xC7k\x85\xF4\x98\xB6\x99<\xCC\xDDY\xFE\x89\x0F@\xFC.\xDA\xDA\xF0\x9D\xC2\x13\xF2>\xDD\x9C=\x8D\x1F`\xC5\v\x9E\xDC\xE7\xECg\xAC)\x06\xA8\x1CD\n\xAA\x9E\xB3m|\xA9\xAD\xFE\x05\xE7\x9D?\x81\xAB\x8A\xFCe_\xDC\xAC\xD6\xCA\x14\xEB\xEFP\xAC~\x94\xE2S\xAAY)>~\x1A\xF9\x1F\x00\x00\xFF\xFF\x03\x00PK\x01\x02\x14\x03\x14\x00\x00\x00\b\x00\eq\x84Ow\xEE\xB7/ \x01\x00\x00\x03\x02\x00\x00\v\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00_rels/.relsPK\x01\x02\x14\x03\x14\x00\x00\x00\b\x00\eq\x84O\xC6\xF5\xA9\xBD`\x01\x00\x00h\x02\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00I\x01\x00\x00DummyProject.DummyPackage.nuspecPK\x01\x02\x14\x03\x14\x00\x00\x00\b\x00Yo\x82O^G\x19\xEB\x02\x06\x00\x00\x00\x10\x00\x00\"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xE7\x02\x00\x00lib/netcoreapp3.0/DummyProject.dllPK\x01\x02\x14\x03\x14\x00\x00\x00\b\x00\eq\x84OA\xDF\xB6Q\xD3\x00\x00\x00\xCB\x01\x00\x00\x13\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00)\t\x00\x00[Content_Types].xmlPK\x01\x02\x14\x03\x14\x00\x00\x00\b\x00\eq\x84O\x1A\xDBV\x92p\x01\x00\x00l\x02\x00\x00Q\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00-\n\x00\x00package/services/metadata/core-properties/eb8aacee7a8446019978a07885ac639d.psmdcpPK\x05\x06\x00\x00\x00\x00\x05\x00\x05\x00\x97\x01\x00\x00\f\f\x00\x00\x00\x00"

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by David Fernandez

Merge request reports