Skip to content

Use Workhorse acceleration on Project Import file upload

Aleksei Lipniagov requested to merge 37256-use-workhorse-acceleration-on-project-import into master

What does this MR do?

Allows using WH accelerated file uploads for Project Import.
This is related to API endpoints only, UI endpoints (Import::GitlabProjectsController) will be accelerated in the next, separate MR.

❗ Connected to gitlab-workhorse!459 (merged) (the change on the WH side, expected to be deployed after this MR)

Since we are touching an existing API, we should deploy it gradually.
The plan suggested by Alessio: gitlab-workhorse!459 (comment 289532237)

How to test locally

  • Make these changes in your local WH (which is in the GDK dir): gitlab-workhorse!459 (merged)
  • Run make gitlab-workhorse-setup && gdk restart gitlab-workhorse to apply them
  • setup minio: brew install minio (if you're on mac)
  • enable object storage and direct upload in config/gitlab.yml
  • uncomment minio start-up command in Procfile (it already has all the keys in the config)
  • restart gdk and make sure you can access http://127.0.0.1:9000
  • you could use mc (minio command-line client) for troubleshooting and for checking storage events
  • for API, I was running curl --request POST --header "PRIVATE-TOKEN: <YOUR_TOKEN>" --form "file=@/Users/lipton/Desktop/tiny-export.tar.gz" http://localhost:3000/api/v4/projects/import --form "path=wh_api_upload"

You will notice in the logs that your request will go through an /authorize endpoint if it was accelerated. Also, headers/body would differ accordingly.

We want to check all three cases locally to verify the change (see the table below).

Conformity

Availability and Testing

We should release overall API change gradually (details in the MR description above).
This MR reflects that approach.

Tested locally with:

Object Storage Off Object Storage On Direct Upload Off Object Storage On Direct Upload On
✅ ✅ ✅

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #37256 (closed)

Edited by 🤖 GitLab Bot 🤖

Merge request reports