Cannot properly set import_url for mirroring - project creation *and* edit through API
Summary
For any project, configuring mirroring by hand is OK, but not through the API.
I discussed about this problem by first contacting Gitlab's EE support.
Steps to reproduce
Here is a piece of python code we use to set mirroring paths (indentation is broken due to copy paste, sorry). Please look carefully at comments inside the code, thank you:
import_url = None
if conf['gitlab'][gitlab].get('mirror_base', False) != False:
import_url = '{}/{}/{}.git'.format(
conf['gitlab'][gitlab]['mirror_base'],
project_ns, # this is a string, not the namespace's id
project_name # the project name as string
)
pe = project_exists(gitlab, project_name, project_ns, project_nsk)
# returns True|False, namespace_id|None, project_id|None
# i manually checked IDs to ensure nothing was wrong, 100% ok
# because those values are taken from the result of the full projects list
# from a gitlab api call. (/projects)
if pe[0] == False:
print(f'==> create project on {gitlab}')
data = {
'name': project_name,
'namespace_id': pe[1],
'description': project['description']
}
if import_url is not None:
data['import_url'] = import_url
r = s.post(f'{url}/projects?private_token={token}', data=data)
elif import_url is not None:
print('==> configure import_url')
data = {
'id': pe[2],
'name': project_name,
'import_url': import_url
}
r = s.put(f"{url}/projects/{pe[2]}?private_token={token}", data=data)
if not r.ok:
raise APIError(r.content)
The conf
var is loaded after this kind of JSON:
{
"gitlab": {
"prod": {
"url": "http://prod.fqdn/api/v4",
"token": "api_access_token",
"clone": "ssh://git@prod.fdqn",
"deploy_key_id": 1
},
"preprod": {
"url": "http://proprod.fdqn/api/v4",
"token": "api_access_token",
"clone": "http://preprod.fqdn",
"mirror_base": "http://user:pass@prod.fqdn",
"deploy_key_id": 3
}
},
"source": "prod",
"namespace": "cool-namespace",
"namespace_kind": "group",
"push_pull": false,
"workdir": "mirror"
}
With this configuration, we simply want to setup mirroring on the gitlab preprod server from (pull) the gitlab prod server.
What is the current bug behavior?
Actually, there are two bugs.
- Creating a project with
import_url
containinguser:pass
in the URL sets the URL, but without authentication informations. - Modifying
import_url
of an existing project doesn't change the import URL. I also tried by removing theuser:pass
in the URL to see if it can be related to authentication informations (parsing error? anything...), but nope, doesn't change anything.
If you want to ask about a bug in our script, be sure i already quadruply checked myself before coming here, the import_url
is actually being modified (or should i say, asked for modification) to the right server and I check to the right server also after :)
Also HTTP status code is always 200
, i checked that with print(r.status_code)
(this is the latest version of python-requests library).
What is the expected correct behavior?
In both cases, import_url
should contain authentication informations.
In the latest case, editing, we should see an update.
Relevant logs and/or screenshots
Started PUT "/api/v4/projects/214?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:25 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:25 +0200
Started POST "/api/v4/projects/215/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:25 +0200
Started PUT "/api/v4/projects/215?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:26 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:26 +0200
Started POST "/api/v4/projects/216/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:26 +0200
Started PUT "/api/v4/projects/216?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:26 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:27 +0200
Started POST "/api/v4/projects/217/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:27 +0200
Started PUT "/api/v4/projects/217?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:27 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:28 +0200
Started POST "/api/v4/projects/218/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:28 +0200
Started PUT "/api/v4/projects/218?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:28 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:28 +0200
Started POST "/api/v4/projects/219/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:28 +0200
Started PUT "/api/v4/projects/219?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:29 +0200
Started GET "/cool-namespace/printers-list/tree/master/files/printers" for 10.110.91.221 at 2017-06-02 11:10:29 +0200
Processing by Projects::TreeController#show as HTML
Parameters: {"namespace_id"=>"cool-namespace", "project_id"=>"printers-list", "id"=>"master/files/printers"}
Completed 200 OK in 83ms (Views: 55.2ms | ActiveRecord: 3.9ms | Elasticsearch: 0.0ms)
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:29 +0200
Started POST "/api/v4/projects/220/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:29 +0200
Started GET "/cool-namespace/printers-list/refs/master/logs_tree/files/printers?_=1496394629604" for 10.110.91.221 at 2017-06-02 11:10:29 +0200
Processing by Projects::RefsController#logs_tree as JS
Parameters: {"_"=>"1496394629604", "namespace_id"=>"cool-namespace", "project_id"=>"printers-list", "id"=>"master", "path"=>"files/printers"}
Completed 200 OK in 31ms (Views: 6.5ms | ActiveRecord: 2.0ms | Elasticsearch: 0.0ms)
Started PUT "/api/v4/projects/220?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:29 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:30 +0200
Started POST "/api/v4/projects/221/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:30 +0200
Started PUT "/api/v4/projects/221?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:30 +0200
Started GET "/cool-namespace/printers-list/tree/master/files/printers" for 10.110.91.221 at 2017-06-02 11:10:30 +0200
Processing by Projects::TreeController#show as HTML
Parameters: {"namespace_id"=>"cool-namespace", "project_id"=>"printers-list", "id"=>"master/files/printers"}
Completed 200 OK in 80ms (Views: 52.8ms | ActiveRecord: 4.2ms | Elasticsearch: 0.0ms)
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:30 +0200
Started POST "/api/v4/projects/222/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:30 +0200
Started GET "/cool-namespace/printers-list/refs/master/logs_tree/files/printers?_=1496394630955" for 10.110.91.221 at 2017-06-02 11:10:31 +0200
Started PUT "/api/v4/projects/222?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:31 +0200
Processing by Projects::RefsController#logs_tree as JS
Parameters: {"_"=>"1496394630955", "namespace_id"=>"cool-namespace", "project_id"=>"printers-list", "id"=>"master", "path"=>"files/printers"}
Completed 200 OK in 35ms (Views: 7.1ms | ActiveRecord: 3.0ms | Elasticsearch: 0.0ms)
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:31 +0200
Started POST "/api/v4/projects/223/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:31 +0200
Started PUT "/api/v4/projects/223?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:31 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:32 +0200
Started POST "/api/v4/projects/224/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:32 +0200
Started PUT "/api/v4/projects/224?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:32 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:32 +0200
Started POST "/api/v4/projects/225/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:33 +0200
Started PUT "/api/v4/projects/225?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:33 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:33 +0200
Started POST "/api/v4/projects/226/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:33 +0200
Started PUT "/api/v4/projects/226?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:33 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:34 +0200
Started POST "/api/v4/projects/227/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:34 +0200
Started PUT "/api/v4/projects/227?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:34 +0200
Started GET "/api/v4/namespaces?private_token=[FILTERED]&search=cool-namespace" for 10.110.90.247 at 2017-06-02 11:10:35 +0200
Started POST "/api/v4/projects/228/deploy_keys/3/enable?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:35 +0200
Started PUT "/api/v4/projects/228?private_token=[FILTERED]" for 10.110.90.247 at 2017-06-02 11:10:35 +0200
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.11.1 Sidekiq Version:5.0.0GitLab information Version: 9.2.2-ee Revision: b004167 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.1 URL: http://ip HTTP Clone URL: http://ip/some-group/some-project.git SSH Clone URL: git@ip:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.0.4 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 5.0.4 ? ... OK (5.0.4) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 5/148 ... repository is empty 9/202 ... repository is empty 9/203 ... ok 9/204 ... ok 9/205 ... ok 9/206 ... ok 9/207 ... ok 9/208 ... ok 9/209 ... ok 9/210 ... ok 9/211 ... ok 9/212 ... ok 9/213 ... ok 9/214 ... ok 9/215 ... ok 9/216 ... ok 9/217 ... ok 9/218 ... ok 9/219 ... ok 9/220 ... ok 9/221 ... ok 9/222 ... ok 9/223 ... ok 9/224 ... ok 9/225 ... ok 9/226 ... ok 9/227 ... ok 9/228 ... ok 9/229 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) projects have namespace: ... 5/148 ... yes 9/202 ... yes 9/203 ... yes 9/204 ... yes 9/205 ... yes 9/206 ... yes 9/207 ... yes 9/208 ... yes 9/209 ... yes 9/210 ... yes 9/211 ... yes 9/212 ... yes 9/213 ... yes 9/214 ... yes 9/215 ... yes 9/216 ... yes 9/217 ... yes 9/218 ... yes 9/219 ... yes 9/220 ... yes 9/221 ... yes 9/222 ... yes 9/223 ... yes 9/224 ... yes 9/225 ... yes 9/226 ... yes 9/227 ... yes 9/228 ... yes 9/229 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.3) Your git bin path is "/opt/gitlab/embedded/bin/git" Git version >= 2.7.3 ? ... yes (2.11.1) Active users: 5
Checking GitLab ... Finished