Adds support to upload and download maven packages from/to GitLab

What does this MR do?

Adds support to upload and download maven packages from GitLab.

• I tested upload with for both mvn deploy and mvn release:perform.
• EEP feature; must be enabled in config
• Uses workhorse file upload
• Should be compatible with object storage

Why was this MR needed?

So people can add GitLab to their pom.xml file and upload/download packages from GitLab

How to test

1. Checkhout this branch
2. Get workhorse changes gitlab-workhorse!283 (comment 91229181)
3. Setup ~/.m2/settings.xml with username and token. See doc in diff for example
4. Create maven project and add GitLab repo to pom.xml. See doc in diff for example

Screenshots (if relevant)

No UI here. UI is in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6781/diffs

Here is terminal output from mvn deploy:

dzaporozhets:~/Projects/mvn-example master 
→ mvn deploy
...
[INFO] --- maven-install-plugin:2.4:install (default-install) @ my-app ---
[INFO] Installing /Users/dzaporozhets/Projects/mvn-example/target/my-app-1.1-SNAPSHOT.jar to /Users/dzaporozhets/.m2/repository/com/mycompany/app/my-app/1.1-SNAPSHOT/my-app-1.1-SNAPSHOT.jar
[INFO] Installing /Users/dzaporozhets/Projects/mvn-example/pom.xml to /Users/dzaporozhets/.m2/repository/com/mycompany/app/my-app/1.1-SNAPSHOT/my-app-1.1-SNAPSHOT.pom
[INFO] 
[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ my-app ---
Downloading from gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/1.1-SNAPSHOT/maven-metadata.xml
Downloaded from gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/1.1-SNAPSHOT/maven-metadata.xml (767 B at 118 B/s)
Uploading to gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/1.1-SNAPSHOT/my-app-1.1-20180806.161455-3.jar
Uploaded to gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/1.1-SNAPSHOT/my-app-1.1-20180806.161455-3.jar (2.6 kB at 3.3 kB/s)
Uploading to gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/1.1-SNAPSHOT/my-app-1.1-20180806.161455-3.pom
Uploaded to gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/1.1-SNAPSHOT/my-app-1.1-20180806.161455-3.pom (2.0 kB at 3.1 kB/s)
Downloading from gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/maven-metadata.xml
Downloaded from gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/maven-metadata.xml (319 B at 809 B/s)
Uploading to gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/1.1-SNAPSHOT/maven-metadata.xml
Uploaded to gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/1.1-SNAPSHOT/maven-metadata.xml (767 B at 1.1 kB/s)
Uploading to gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/maven-metadata.xml
Uploaded to gitlab-maven: http://localhost:3001/api/v4/projects/1/packages/maven/com/mycompany/app/my-app/maven-metadata.xml (319 B at 50 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17.331 s
[INFO] Finished at: 2018-08-06T19:15:04+03:00
[INFO] Final Memory: 16M/55M
[INFO] ------------------------------------------------------------------------

Does this MR meet the acceptance criteria?

• Changelog entry added, if necessary
• Documentation created/updated
• API support added
• Tests added for this feature/bug
• Conforms to the code review guidelines
  • [-] Has been reviewed by a UX Designer
  • [-] Has been reviewed by a Frontend maintainer
  • Has been reviewed by a Backend maintainer
  • Has been reviewed by a Database specialist
• EE specific content should be in the top level /ee folder
• Conforms to the merge request performance guidelines
• Conforms to the style guides
• Conforms to the database guides
• If you have multiple commits, please combine them into a few logically organized commits by squashing them
• [-] Internationalization required/considered
• If paid feature, have we considered GitLab.com plan and how it works for groups and is there a design for promoting it to users who aren't on the correct plan
• End-to-end tests pass (package-and-qa manual pipeline job)

What are the relevant issue numbers?

Closes #5811 (closed)

added 1 commit

• cb9f4709 - Generate models necessary for maven repository backend

Compare with previous version

assigned to @dzaporozhets

changed milestone to %11.3

added 1 commit

• 0809d2cd - Add associations and cascade delete for foreign keys

Compare with previous version

added 1 commit

• a6ddc13d - Add package uploader [ci skip]

Compare with previous version

added 1 commit

• 506eff26 - Add packages to config files [ci skip]

Compare with previous version

added 1 commit

• 5f425c73 - Make package migrations pass [ci skip]

Compare with previous version

added 1 commit

• b2d34209 - Add support for maven package download via API [ci skip]

Compare with previous version

download works with repo section in pom.xml :

  <repositories>
    <repository>
      <id>gitlab-maven</id>
      <url>http://localhost:3001/api/v4/projects/8/maven</url>
    </repository>
  </repositories>

Logs

→ mvn install
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Building o-my-app 1.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading from gitlab-maven: http://localhost:3001/api/v4/projects/8/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/maven-metadata.xml
Downloaded from gitlab-maven: http://localhost:3001/api/v4/projects/8/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/maven-metadata.xml (767 B at 53 B/s)
Downloading from gitlab-maven: http://localhost:3001/api/v4/projects/8/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/my-app-1.0-20180724.124855-1.pom
Downloaded from gitlab-maven: http://localhost:3001/api/v4/projects/8/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/my-app-1.0-20180724.124855-1.pom (1.6 kB at 7.0 kB/s)
Downloading from gitlab-maven: http://localhost:3001/api/v4/projects/8/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/my-app-1.0-20180724.124855-1.jar
Downloaded from gitlab-maven: http://localhost:3001/api/v4/projects/8/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/my-app-1.0-20180724.124855-1.jar (2.5 kB at 11 kB/s)

added 1 commit

• 6769b9d0 - Add minimal UI for project packages [ci skip]

Compare with previous version

added 1 commit

• 82b186e7 - Refactor maven repository API

Compare with previous version

added 1 commit

• 19e46343 - Implement upload package logic without workhorse for now

Compare with previous version

	1 Warning
	This merge request is quite big (more than 1142 lines changed), please consider splitting it into multiple merge requests.

	1 Message
	This merge request adds or changes files that require a review from the Database team.

Database Review

The following files require a review from the Database team:

• ee/db/migrate/20180720120716_create_packages_packages.rb
• ee/db/migrate/20180720120726_create_packages_package_files.rb
• ee/db/migrate/20180720121404_create_packages_maven_metadata.rb
• db/schema.rb

To make sure these changes are reviewed, take the following steps:

1. Edit your merge request, and add gl-database to the list of Group approvers.
2. Mention @gl-database in a separate comment, and explain what needs to be reviewed by the team. Please don't mention the team until your changes are ready for review.

Generated by Danger

added 2 commits

• c1e36186 - Require auth for non get requests to maven packages
• 28825f40 - Make package uploading work and add some docs about feature

Compare with previous version

added 516 commits

• 28825f40...63e4129d - 505 commits from branch master
• 9ce56604 - Generate models necessary for maven repository backend
• 085c5f76 - Add associations and cascade delete for foreign keys
• 4475867a - Add package uploader [ci skip]
• d72acf6b - Add packages to config files [ci skip]
• 9b426f4b - Make package migrations pass [ci skip]
• 65794f9a - Add support for maven package download via API [ci skip]
• b5f55f39 - Add minimal UI for project packages [ci skip]
• 7ed98216 - Refactor maven repository API
• e769b3d6 - Implement upload package logic without workhorse for now
• 35a7c221 - Require auth for non get requests to maven packages
• 76fd0dac - Make package uploading work and add some docs about feature

Compare with previous version

added 1 commit

• 5addd652 - Refactor maven packages code [ci skip]

Compare with previous version

added 1 commit

• 11552b10 - Fix wrong file serve when deploy new maven package build with same version [ci skip]

Compare with previous version

mentioned in merge request gitlab-workhorse!283 (merged)

added 2 commits

• d369908b - Add more UI to browse maven packages [ci skip]
• 38d8b47f - Use workhorse for maven package file upload [ci skip]

Compare with previous version

@nolith I have some trouble making it work. Ruby changes are https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6607/diffs?commit_id=38d8b47f68a85e1fd6c9ba9376c7c5a1227febe3. It hits authorize, and then original request which means your workhorse change is working. However when the request is passed to Ruby I don't see any workhorse related info in params to be later used in UploadedFile.from_params

Logs for request:

Started PUT "/api/v4/projects/8/packages/maven/com/mycompany/app/my-app/1.3-SNAPSHOT/my-app-1.3-20180730.170814-3.jar/authorize" for 127.0.0.1 at 2018-07-30 20:08:14 +0300
  PersonalAccessToken Load (0.2ms)  SELECT  "personal_access_tokens".* FROM "personal_access_tokens" WHERE "personal_access_tokens"."token" = $1 LIMIT 1  [["token", "78NbnPqnyRQop3Tqzgj2"]]
  User Load (0.5ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 2]]
  License Load (0.2ms)  SELECT  "licenses".* FROM "licenses"  ORDER BY "licenses"."id" DESC LIMIT 1

DEBUG RESPONSE {:TempPath=>"/Users/dzaporozhets/Projects/gdk-ee/gitlab/shared/packages/tmp/uploads"}


Started PUT "/api/v4/projects/8/packages/maven/com/mycompany/app/my-app/1.3-SNAPSHOT/my-app-1.3-20180730.170814-3.jar" for 127.0.0.1 at 2018-07-30 20:08:14 +0300
  PersonalAccessToken Load (0.2ms)  SELECT  "personal_access_tokens".* FROM "personal_access_tokens" WHERE "personal_access_tokens"."token" = $1 LIMIT 1  [["token", "78NbnPqnyRQop3Tqzgj2"]]
  User Load (0.5ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 2]]
  License Load (0.2ms)  SELECT  "licenses".* FROM "licenses"  ORDER BY "licenses"."id" DESC LIMIT 1

DEBUG PARAMS {"id"=>"8", "app_group"=>"com/mycompany/app", "app_name"=>"my-app", "app_version"=>"1.3-SNAPSHOT", "file_name"=>"my-app-1.3-20180730.170814-3.jar"}

@dzaporozhets from

DEBUG RESPONSE {:TempPath=>"/Users/dzaporozhets/Projects/gdk-ee/gitlab/shared/packages/tmp/uploads"}

I'll say that direct_upload is disabled

Yes, it is disabled.

It seems we need a bit more changes to workhorse to make it work. Because the way maven sends files is different from the way we do in GitLab. @nolith will make those and we can proceed with it.

changed the description

changed title from WIP: Resolve "Add Maven support to our Artifact Repository (MVC)" to WIP: Adds support to upload and download maven packages from GitLab

changed title from WIP: Adds support to upload and download maven packages from GitLab to WIP: Adds support to upload and download maven packages from/to GitLab

@sarrahvesselov can someone from UX look into this feature? It is mostly backend, but I think users will appreciate some basic UI here.

From my perspective I would like to have:

• Ability to see uploaded packages
• Ability to see whats inside a package
• Ability to remove a package

This is what I have so far:

I still need to figure out where to put this Packages page into our navigation. The feature by idea is pretty similar to our container registry.

@dzaporozhets workhorse MR has been updated.

[INFO] --- maven-deploy-plugin:2.7:deploy (default-deploy) @ my-app ---
Downloading from local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/maven-metadata.xml
Uploading to local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/my-app-1.0-20180801.104005-1.jar
Uploaded to local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/my-app-1.0-20180801.104005-1.jar (2.4 kB at 1.1 kB/s)
Uploading to local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/my-app-1.0-20180801.104005-1.pom
Uploaded to local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/my-app-1.0-20180801.104005-1.pom (1.0 kB at 782 B/s)
Downloading from local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/maven-metadata.xml
Uploading to local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/maven-metadata.xml
Uploaded to local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/1.0-SNAPSHOT/maven-metadata.xml (767 B at 532 B/s)
Uploading to local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/maven-metadata.xml
Uploaded to local-gitlab: http://localhost:3001/api/v4/projects/15/packages/maven/com/mycompany/app/my-app/maven-metadata.xml (281 B at 155 B/s)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 54.646 s
[INFO] Finished at: 2018-08-01T12:40:12+02:00
[INFO] ------------------------------------------------------------------------

Sure thing @dzaporozhets. @dimitrieh or @matejlatin, do either of you have a moment to take a look at this? If not, let me know and I will see if anyone else is available.

@nolith perfect, it works for me too. Now I will work on addressing https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6607#note_91590788

I believe gitlab-workhorse!283 (merged) can be merged independently from this MR, right?

I believe gitlab-workhorse!283 (merged) can be merged independently from this MR, right?

@dzaporozhets I think so. It's a new path so there should be no problems.

@dzaporozhets initial comments:

• where is this page located in the applicaton (how can you navigate towards it?)
• upload/download (should there be buttons + modals for these actions?)
• I think I would expect a delete button on the table view as well (at the right end, similar to environments or variables for example)
• For the tables on the detail view, it might be nice to consistently have the data headers to be left aligned and the data itself to be right aligned

We want to add Maven support to our Artifact repository, to make it easier to work with for Java based projects/developers.

Can you detail this a little bit more, how this mvc helps this?

From my perspective I would like to have:

• Ability to see uploaded packages

• Ability to see whats inside a package

• Ability to remove a package

Like how does this solve the problem?

added 1 commit

• d5ed1242 - Specify namespace for PackageFileUploader [ci skip]

Compare with previous version

added 1 commit

• 0dd5e9c1 - Add packages to navigation and policies [ci skip]

Compare with previous version

added 1 commit

• 6d864df5 - Add some basic feature specs on UI [ci skip]

Compare with previous version

TODO:

1. Write API specs
2. Move code to EE dir
3. Add feature specs for remiving package
4. Refactor the API to avoid reading uploaded file to absolute minimum. Make it compatible with object storage
5. More docs!

added 1 commit

• fec8bd2c - Add specs for maven packages api [ci skip]

Compare with previous version

This MR is already 900 lines of code so I probably move UI to separate merge request.

added 1 commit

• 0b53e667 - Add specs for maven packages api [ci skip]

Compare with previous version

added 5 commits

• f1863e94 - Move packages specs and API to ee/ directory
• 70158cb8 - Move packages migrations to ee/
• d4a72d40 - Move more package files to ee/ directory
• 32036115 - Remove UI related code for packages [ci skip]
• d5c6be46 - Move package uploader to ee dir [ci skip]

Compare with previous version

added 1 commit

• 5f595991 - Remove packages tab from sidebar

Compare with previous version

mentioned in merge request !6781 (merged)

MR with UI => https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6781

added 611 commits

• 5f595991...278d3943 - 586 commits from branch master
• 10372461 - Generate models necessary for maven repository backend
• 897c627f - Add associations and cascade delete for foreign keys
• 23d154ab - Add package uploader [ci skip]
• bcbf4260 - Add packages to config files [ci skip]
• 6518d67b - Make package migrations pass [ci skip]
• c2c42054 - Add support for maven package download via API [ci skip]
• c7993297 - Add minimal UI for project packages [ci skip]
• 9e5db89a - Refactor maven repository API
• 6929c480 - Implement upload package logic without workhorse for now
• 71022539 - Require auth for non get requests to maven packages
• 47014f78 - Make package uploading work and add some docs about feature
• a5ef4a97 - Refactor maven packages code [ci skip]
• 16b16ae3 - Fix wrong file serve when deploy new maven package build with same version [ci skip]
• 4528b093 - Add more UI to browse maven packages [ci skip]
• 31307c52 - Use workhorse for maven package file upload [ci skip]
• df4e3c17 - Specify namespace for PackageFileUploader [ci skip]
• 2517abef - Add packages to navigation and policies [ci skip]
• 96043159 - Add some basic feature specs on UI [ci skip]
• bc924e0a - Add specs for maven packages api [ci skip]
• ef00657d - Move packages specs and API to ee/ directory
• eb1f5dff - Move packages migrations to ee/
• 00f76080 - Move more package files to ee/ directory
• 91ac9c8a - Remove UI related code for packages [ci skip]
• 282ed809 - Move package uploader to ee dir [ci skip]
• e967f657 - Remove packages tab from sidebar

Compare with previous version

TODO:

1. API check for permissions to download/upload package
2. Refactor the API to avoid reading uploaded file to absolute minimum. Make it compatible with object storage
3. More docs!
4. Tasks in https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6781#note_92316255

added 1 commit

• 17c9b2e1 - Make package migrations reversible

Compare with previous version