Skip to content

Resolve "Imports fail in 14.5.2 fail with HTTParty::UnsupportedURIScheme error"

Illya Klymov requested to merge 349096-imports-fail-in-14-5-2-fail-with-httparty-unsupportedurischeme-error into master

What does this MR do and why?

!70165 (merged) introduced validation for git URI. This validation was intended to save error budgets for our importers when user is providing incorrect URI and provide nice feedback on frontend before user even starts their import

However, in !70165 (merged) we forgot about git:// protocol Unfortunately both v2 and v0 (there is no v1 git protocol in the wild) are problematic for such validation

In http(s) we can parse just first message line to be sure, that this at least appears to be valid git repository. Unfortunately, git:// protocol is command oriented and the closest possible solution is to issue git ls-remote request and parse entire output if it is correct. It is problematic both in terms of implementation (parsing binary data which is coming via socket, yay) and questionable from security perspective - since we need to store entire response in memory before parsing (or implement streaming parser) it becomes quite easy to abuse this one

This MR uses most boring solution - we simply allow all git:// URLs without validation - exactly the same behavior we had before !70165 (merged) was merged

While this will potentially increase error rate - amount of git:// imports is relatively low, and it unblocks our customers

Related to #349096 (closed)

video

Projects___Dashboard___GitLab

How to set up and validate locally

  1. Find a directory, where you have at least one git repository stored (don't use gitlab for this one, it is very big and cloning via git daemon is not the best idea). Assume repository is foo
  2. Run touch foo/.git/git-daemon-export-ok to allow git daemon to export repo
  3. Run git daemon --reuseaddr --base-path=. . (note: this should be run not inside repo directory, one directory higher in fs hierarchy)
  4. Make sure your gitlab instance allow localhost requests (Admin > Settings > Network)
  5. Use /projects/new#import_project -> Import by URL
  6. Use URL git://127.0.0.1/foo
  7. Observe import completes successfully

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Illya Klymov

Merge request reports