Skip to content

Permit special characters (#, / and ?) in credentials provided in import_url when creating/updating a project via the API

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When creating a new project via the API and specifying import_url, the presence of special characters like #, / and ? in the password yield an error like:

Screen_Shot_2021-10-20_at_11.13.16_AM

EDIT: A similar situation can happen when editing a project via the API and specifying import_url containing special characters. See #343452 (comment 977233558) for more details.

Steps to reproduce

  1. Issue an API call like the one shown below
  2. Observe an error like the one shown below

API Call

curl --request POST \
  --url 'https://gitlab.example.com/api/v4/projects?name=delta&import_url=https://cutecat:apassword#@gitlab.example.com/hello/real-secrets.git&namespace_id=35' \
  --header 'Authorization: Bearer ATOKEN' \

Response

{
  "message": {
    "import_url": [
      "is blocked: URI is invalid"
    ]
  }
}

Example Project

What is the current bug behavior?

A is blocked: URI is invalid error is returned.

What is the expected correct behavior?

The error should not be returned, the API call should proceed and a new project should be created (provided all other info in the API call is correct and accurate).

Relevant logs and/or screenshots

At the GitLab Rails console, we can observe:

irb(main):023:0> Gitlab::UrlBlocker.validate!  "https://cutecat:apassword#@gitlab.example.com/hello/real-secrets.git"

Traceback (most recent call last):
        4: from (irb):23
        3: from lib/gitlab/url_blocker.rb:37:in `validate!'
        2: from lib/gitlab/url_blocker.rb:166:in `parse_url'
        1: from lib/gitlab/url_blocker.rb:171:in `rescue in parse_url'
Gitlab::UrlBlocker::BlockedUrlError (URI is invalid)
Workarounds
  • Use a token instead.
  • Change the password so that it does not contain the #, / or ? characters.

The following (non-exhaustive) list of characters are permitted:

  • \
  • !
  • `
  • ~
  • {
  • %
  • |

Output of checks

This happens on 14.3.3 and 13.3.5.

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)


#  gitlab-rake gitlab:env:info

System information
System:		Ubuntu 18.04
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.4p191
Gem Version:	3.1.4
Bundler Version:2.1.4
Rake Version:	13.0.6
Redis Version:	6.0.14
Git Version:	2.33.0.
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	14.3.3-ee
Revision:	6d764ffb32f
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.7
URL:		https://test.brie.fqdn
HTTP Clone URL:	https://test.brie.fqdn/some-group/some-project.git
SSH Clone URL:	git@test.brie.fqdn:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:

GitLab Shell
Version:	13.21.1
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git


(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


~# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.21.1 ? ... OK (13.21.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
not verifying SSL hostname of LDAPS server 'ldap.FQDN.TLD:389'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 8 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
2/1 ... yes
12/2 ... yes
12/3 ... yes
12/4 ... yes
12/5 ... yes
12/6 ... yes
12/7 ... yes
12/8 ... yes
13/9 ... yes
13/10 ... yes
13/11 ... yes
13/12 ... yes
14/13 ... yes
14/14 ... yes
14/15 ... yes
15/16 ... yes
15/17 ... yes
15/18 ... yes
15/19 ... yes
1/22 ... yes
16/24 ... yes
19/25 ... yes
1/26 ... yes
1/27 ... yes
1/28 ... yes
1/29 ... yes
1/30 ... yes
1/31 ... yes
16/32 ... yes
21/33 ... yes
16/34 ... yes
23/35 ... yes
22/36 ... yes
22/37 ... yes
22/38 ... yes
22/39 ... yes
22/40 ... yes
22/41 ... yes
22/42 ... yes
22/43 ... yes
25/44 ... yes
25/45 ... yes
25/46 ... yes
25/47 ... yes
2/48 ... yes
28/49 ... yes
30/50 ... yes
31/51 ... yes
32/52 ... yes
22/53 ... yes
34/54 ... yes
33/55 ... yes
36/56 ... yes
35/57 ... yes
35/58 ... yes
37/59 ... yes
35/60 ... yes
35/61 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.4)
Git version >= 2.31.0 ? ... yes (2.33.0)
Git user has default SSH configuration? ... yes
Active users: ... 18
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

A few items that are somewhat related that I reviewed in the process of looking into this:

Edited by 🤖 GitLab Bot 🤖